Download as PDF, PPTX
![Global average broadband
57.91
Download
Mbs
28.68
Upload
Mbs
Upload time =
data volume
rate
= 5000000000[MB] x 8[Mb/MB]
28.68 Mbs
=
44 years!!!!](https://image.slidesharecdn.com/jonthebeach-190522125601/75/The-big-data-Universe-Literally-9-2048.jpg)
![Global average broadband
57.91
Download
Mbs
28.68
Upload
Mbs
Upload time =
data volume
rate
= 5000000000[MB] x 8[Mb/MB]
28.68 Mbs
=
44 years!!!!](https://crownmelresort.com/image.slidesharecdn.com/jonthebeach-190522125601/75/The-big-data-Universe-Literally-9-2048.jpg)
Uploaded byJ On The Beach
The big data Universe. Literally.
The document discusses the challenges of big data in astronomy. It describes several upcoming and current astronomical surveys that will produce enormous amounts of data, such as Euclid which will collect 15 terabytes per day starting in 2022. Dealing with such large datasets requires new techniques for storage, processing, analyzing, classifying, compressing and visualizing the data. Machine learning and citizen science are helping to process the data more efficiently. Future surveys like the Square Kilometer Array telescope will pose even greater challenges by producing data at terabytes per second.
More Related Content
Similar to The big data Universe. Literally.
The big data Universe. Literally.
- 1. The Dark Matter Mystery The big datauniverse. Literally D r M a g g i e L i e u @space_mog The big data universe. Literally Bruno Merin & Beatriz Martinez
- 2.
- 3.
- 4.
- 5. A telescope thesize of the Earth! Event Horizon Telescope EHT - Radio telescopes from around the world
- 6. 📸 : FloraGraham 5PB of data
- 7. 1 blackhole =5000TB 1TB/day 8TB/day
- 8. Global average broadband DownloadUpload 57.91Mbs 28.68Mbs
- 9. Global average broadband 57.91 Download Mbs 28.68 Upload Mbs Uploadtime = data volume rate = 5000000000[MB] x 8[Mb/MB] 28.68 Mbs = 44 years!!!!
- 10.
- 11. 📸 : MITHaystack Observatory
- 12. 📸 : KatieBouman 2 years of processing
- 13. But space datais different….
- 17. ‣ coming soonin 2022 Euclid
- 19.
- 20.
- 21. foreground mass More mass= more distorted galaxies!
- 22. real galaxy shear atmosphere& telescope blur pixelised by detectors noise photo cred: CFHT
- 24. Euclid Wide Survey 0° 30° 60° -30° -60° 315°0°45°270° 225°90°135° Moon for comparison ‣ 15,000 deg2
- 25. Flagship simulations ‣End-2-end simulations ‣ Largest simulated galaxy catalogue ever built ‣ 2 trillion dark matter particles
- 26. ‣ Swiss NationalComputing Centre - 6th fastest computer in the world ‣ 80hrs ‣ 270,000 EUR Piz Daint - over 5000 GPU nodes
- 27. ‣ 5000 deg2 ‣Raw simulation data: 0.4PB ‣ Compress to catalogs ‣ Rockstar (5.5TB) ‣ 2D dark matter count maps (1TB)
- 28.
- 29. DSA-2 DSA-3 ESA-ESOC ‣ ESA deepspace tracking stations DSA-1
- 30. ‣ Ground station:Cebreros, Spain ‣ 4 hr communication window ‣ Steerable K-band (26 GHz) ‣ X-band (8.5 GHz) ‣ Data rate: 850 Gbit/day ‣ On board 4Tbit flash memory Telemetry
- 31. ‣ Science centre:ESAC, Spain ‣ Ex-telemetry, tracking & commanding station ‣ Quick look analysis, archiving and distribution
- 32.
- 33. Attractive data No longerera where we fight for data, but era that we choose data!
- 34. Visualising the data ‣ESA sky ‣ HiPS map, based on HEALpix ‣ Visualise TB’s of data ‣ Render kB of data Fernique et al 2015
- 35.
- 36. Code-to-data platform ‣ ScienceExploration platform: ‣ Jupyter notebooks on SPARK clusters ‣ Cloud computing: ‣ Amazon web services, ‣ Google cloud, etc
- 37. See Lieu+18 ‣ Rawdata is nasty! ‣ with GB’s data per day, traditional methods are not efficient Analysing the data
- 38.
- 39. Classification and detection ‣K-means Characterising spectra of galaxies, Rahmani+18
- 40. Classification and detection ‣DBSCAN core core border noise Stars in a star forming region, Canovas+19
- 41. Classification and detection ‣K-nearest neighbour Defining types of supernovae, Lochner+16
- 42. Classification and detection ‣Decision tree T > 2 S > 5 F>0. 3 D CB A T F T T FF Finding weird galaxies, Baron+16
- 43. Classification and detection ‣Convolutional neural networks (& object detection) Strong gravitational lensing, Schaefer+17 Asteroids, Lieu+19
- 44. Classification and detection ‣Transfer learning
- 45. Freeze Replace Classification and detection ‣Transfer learning
- 46. Citizen science ‣ Outsourcetasks to the general public ‣ Zooniverse platform: easy to build projects ‣ 100’s of Projects ‣ 250M classifications ‣ 2M Volunteers
- 47. Citizen science Classify galaxymorphologies: Lintott+08
- 48. Citizen science Find starforming bubbles: Kendrew+12
- 49. Citizen science Discover glitchesin gravitational wave signals: Zevity+17
- 50. Data & modelcompression ‣ Neural networks & emulators Emulate the halo mass function with mixture density networks, Lieu+in. prep Emulate cosmology with neural density estimators, Alsing+2019 Scaling relations with principle component analyses PCA, Bothwell+2016
- 51. ‣ Edge computing: ‣Some form in Gaia ‣ Mars rovers Upcoming methods
- 52. ‣ Continuous learningwith GANS Upcoming methods
- 53. ‣ Federated learning Upcomingmethods Super Model ESAC Data
- 54. 10PB of datais nothing…
- 56. ‣ First light:2022 ‣ Duration: 10 years ‣ Data: 15TB/day
- 57. ‣ Search fortransients (supernova, asteroids, comets, gamma ray bursts) ‣ Gravitational lensing img: Pursiainen
- 58. Can it getany worse…?
- 59. ‣ SKA -square kilometre array ‣ First light: 2030 ‣ Data: 2TB/sec
- 60. Karoo, South Africa SKA-Mid MRO,Australia SKA-Low
- 61.
- 62.
- 63.
- 64.
- 65. Biggest challenges inAstronomy ‣ Collecting the data ‣ Retrieval ‣ Filtering good from bad ‣ Data Storage ‣ Distributing the data ‣ Upload/Download ‣ Combining data ‣ complementary observations and multi-wavelength observations ‣ Data analysis ‣ Compression ‣ Source detection ‣ Visualising the data Let’s collaborate!
- 66. Thank you foryour attentioN