# Dallas Google PoP ## New Google POP ``` Site DFW 28 Cage 4000.40 1950 N Stemmons Freeway Suite 4000 Dallas, TX 75207 Racks 4-11 4-12 RGnet in 4-12 IETF/Deadalus/RGnet in 4-11 Shipping Address: Google - chris morrow c/o: Infomart Suite 4000 1950 N Stemmons Freeway Dallas, TX 75207 ``` ## Rack Physical and Wiring | hostname | switch | ipmi | oob | oobname | pwr | rack | | | :--------------------------- | ---------------------| ------------ | ------- | ---------- | ---------------------------| ----- | ------------------------------------- | | o2.dfw.rg.net | sw0.35 | | 1.8 | o2 | | | Cyclades® ACS Advanced Console Server | | [equnix IX fiber] | 0.46 | | | | | | | | r1.dfw.rg.net.2/2/0 | 0.07 | | 2.15 | r1 | | 0.13-17 | LAN | | r1.dfw.rg.net.2/2/1 | 2.03 | | 2.15 | r2 | | 0.13-17 | NTT (VLAN 53) | | r2.dfw.rg.net-0/0/1/0 | 2.05 | 0.08 | 2.3 | r2 | | 0.18.19 | NTT VLAN 50 | | r2.dfw.rg.net.0/1/1 | 2.06 | | 2.3 | | | 0.18.19 | LAN | | o1.dfw.rg.net | 0.40 | | 0.aux | | 1.1 | 0.22-23 | | | sw0.dfw.rg.net | 2.47 | | 2.13 | sw0 | [^1] | 0.28 | | | sw2.dfw.rg.net | 0.47 | | 2.02 | sw2 | [^1] | 0.30 | | | vmX.dfw.rg.net | [^1] | 2.31 | 1.10 | vm0 | 0.4/1.4 | 0.31-32 | | | raid0.dfw.rg.net | 2.16 | 2.18 | 2.44 | raid0 | 0.5/1.5 | 0.36-38 | | | vm0.dfx.rg.net | 2.27 2.25(drbd) | 2.20 | 2.5 | vm0.dfx | pwr0b B1-9 pwr0r B1-9 | 0.40 | | | vm1.dfx.rg.net | 2.28 2.26(drbd) | 2.19 | 2.6 | vm1.dfx | pwr0b B1-10 pwr0r B1-10 | 0.41 | | | dfw0.psg.com | 2.13 | 2.12 | 2.12 | dfw0 | 0.3/1.3 | 0.43 | | | nas1.sql1.ietf.org | [^1] | 2.11 | 2.11 | vm3 | 0.4/1.4 | 1.17 | | | vm2.dfw.rg.net | [^1] | 2.33 | 2.43 | vm2 | 0.4/1.4 | 1.18-19 | | | vm1.dfw.rg.net | [^1] | 2.32 | 2.42 | vm1 | 0.4/1.4 | 1.20.21 | | | vm0.dfw.rg.net | [^1] | 2.31 | 2.41 | vm0 | 0.4/1.4 | 1.22.23 | | | b0.noc.ietf.org | [^1] | 2.34 | 2.45 | vm3 | 0.4/1.4 | 1.26 | | | sw3.dfw.rg.net | [^1] | 2.34 | 2.30 | sw3 | 0.4/1.4 | 1.30 | | | dallas01.mux.peering.usc.edu | 0.38 | | 2.46 | usc | 0.7/1.7 | 1.40 | | | rtcolo.daedalus.com | | xx | 2.45 | b0 | xx | 1.41 | | | paddington.richosman | 0.22 | | 2.47 | paddington | 0.8/1.8 | 1.42 | | | kingsx.richosman.com | 0.21 | | 2.48 | kingsx | 0.6/1.6 | 1.43-44 | | | pwr1.red.dfw.rg.net | 0.41 | | 2.9 | pwr1-red | subfl | 26 | | | pwr1.blue.dfw.rg.net | 0.41 | | 2.10 | pwr1-blue | subfl | 26 | | | pwr0-blue.dfw.rg.net | 0.42 | | 2.16 | pwr0-blue | subfl | - | | | pwr0-red.dfw.rg.net | 0.42 | | 2.17 | pwr0-red | subfl | - | | Removed | hostname | switch | ipmi | oob | oobname | pwr | rack | | | :--------------------------- | ---------------------| ------------ | ------- | ---------- | ---------------------------| ----- | ------------------------------------- | | r0.dfw.rg.net-rp0 | 0.45 | | 0.01 | r00 | subfl | 03-13 | | | r0.dfw.rg.net-rp0mgt | 0.33 | | | | | 03-13 | | | r0.dfw.rg.net-rp1 | 0.?? | | 0.02 | r01 | | 03-13 | | | r0.dfw.rg.net-rp1mgt | 0.34 | | | | | 03-13 | | | [r0.2/0/4 LAN] | 0.04 | | | | | 03-13 | | | [r0.2/0/5 ntt fiber] | 0.05 | | | | | 03-13 | | | [r0.2/0/6 spare] | 0.06 | | | | | 03-13 | | | [r0.2/0/7 equinix ix] | 0.15 | | | | | 03-13 | | | sw1.dfw.rg.net | | | 0.13 | sw1 | [^1] | 0.35 | | [^2]: PTSS/B is connected to power 0.2/1.2 and controls both sw0 and sw1 ## Ganeti Cluster / 198.180.152.30/31/32/33 ``` vm0.dfx.rg.net:/home/randy> sudo gnt-node list Node DTotal DFree MTotal MNode MFree Pinst Sinst vm0.dfx.rg.net 3.1T 1.6T 251.8G 197.3G 232.2G 6 3 vm1.dfx.rg.net 3.1T 1.9T 251.8G 26.1G 241.0G 6 6 ``` ### vm0.dfx.rg.net vm1.dfx.rg.net VMs are [Supermicro A+ Server 1114S-WTRT Systems](https://www.supermicro.com/Aplus/system/1U/1114/AS-1114S-WTRT.cfm)
With [Supermicro H12SSW-NT Motherboards](https://www.supermicro.com/en/products/motherboard/H12SSW-NT)
With one AMD EPYCTM 7443P Single Socket Processor 24-Core 2.85GHz
With 256G of RAM each
And 2 Samsung 250GB 970 EVO+ Series - M.2 NVMe PCIe (<1DWPD) (MZ-V7S250B)
And 8 Samsung 960GB 883 DCT Series - 2.5" SSD SATA 6Gb/s (<1DWPD) (MZ-7LH960NE)
Serial Numbers: iX1210A-IXN S/N's A1-86827 and A1-86828
[^1]: vm servers' ethers are connected to sw2.dfw as follows: | vm | ipmi | e0 | e1 | | ---- | ------ | ------ | ------ | | vm0 | sw2-20 | sw2-25 | sw2-26 | | vm1 | sw2-19 | sw2-27 | sw2-28 | ### vm[0-1].dfx.rg.net Disk Drives ``` vm0.dfx.rg.net:/home/randy# for i in a b c d e f g h; do echo -n "sd$i "; smartctl -a /dev/sd$i | egrep "Serial|Model"; done sda Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316039 sdb Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R107119 sdc Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316036 sdd Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316032 sde Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R303811 sdf Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316042 sdg Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R107124 sdh Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316040 vm1.dfx.rg.net:/home/randy# for i in a b c d e f g h; do echo -n "sd$i "; smartctl -a /dev/sd$i | egrep "Serial|Model"; done sda Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316035 sdb Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R303827 sdc Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316034 sdd Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R303829 sde Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R303777 sdf Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R303803 sdg Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R303764 sdh Device Model: SAMSUNG MZ7LH960HAJR-00005 Serial Number: S45NNA0R316031 ``` ### Hosts on vm0/1.dfw.rg.net | Instance | Primary_node | Secondary_Node | MaxMem | DiskUsage | IP_Address | SysAdmin | | ------------------- | -------------- | -------------- | ------ | --------- | --------------- | ------------------------------------------ | | ca.rg.net | vm0.dfw.rg.net | vm1.dfw.rg.net | 4.0G | 128.0G | 198.180.152.6 | Randy Bush | | cu0.dfw.rg.net | vm1.dfw.rg.net | vm0.dfw.rg.net | 512M | 12.0G | 198.180.152.71 | Steven M Bellovin John Koh | | git.rg.net | vm0.dfw.rg.net | vm1.dfw.rg.net | 2.0G | 34.0G | 198.180.152.8 | Randy Bush | | icir.dfw.rg.net | vm0.dfw.rg.net | vm1.dfw.rg.net | 1.0G | 32.0G | 198.180.152.57 | Bill Marczak | | mon0.rg.net | vm1.dfw.rg.net | vm0.dfw.rg.net | 16G | 150.1G | 198.180.152.62 | Randy Bush | | netsrv.dfw.rg.net | vm0.dfw.rg.net | vm1.dfw.rg.net | 1.0G | 100G | 198.180.152.15 | Randy Bush | | nms.rg.net | vm0.dfw.rg.net | vm1.dfw.rg.net | 2.0G | 20G | 198.180.152.9 | Randy Bush | | petunia.hactrn.net | vm0.dfw.rg,net | vm1.dfw.rg.net | 6G | 120G | 198.180.152.200 | Rob Austein | | wp.rg.net | vm1.dfw.rg.net | | 4G | 64G | 198.180.152.56 | Randy Bush | | ymbk.com | vm1.dfw.rg.net | vm0.dfw.rg.net | 512M | 16.1G | 198.180.152.10 | Randy Bush | ## raid0.dfw.rg.net [Mobo Supermicro X7SB4/E](https://www.supermicro.com/products/motherboard/Xeon3000/3210/X7SB4.cfm) [IPMI AOC-IPMI20-E](http://www.supermicro.com/products/accessories/addon/AOC-IPMI20-E.cfm) ### raid0 disk drive physical order markings on drive carriers | | | | | | ---- | ---- | ---- | ---- | | 00 | 10 | 20 | 30 | | 01 | 11 | 21 | 31 | | 02 | 12 | 22 | 32 | | 03 | 13 | 23 | 33 | ``` for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do echo -n "1/$i "; smartctl -i -d hpt,1/$i /dev/hptrr | grep ^Serial; done ``` 1/1 Serial Number: WD-WMAZA9378538 1/2 Serial Number: V1KVXSXG 1/3 Serial Number: YGGU1YTA 1/4 Serial Number: WD-WMAZA9520754 1/5 Serial Number: WD-WCAVY0114154 1/6 Serial Number: WD-WMC5C0D9L6HV 1/7 Serial Number: WD-WCC6N5UY8H5P 1/8 Serial Number: WD-WCC300759632 1/9 Serial Number: WD-WCC6N6DDJ7DT 1/10 Serial Number: WD-WCC6N6VTE2NF 1/11 Serial Number: WD-WMC5C0D8Z7FV 1/12 Serial Number: WD-WMC6N0J6S9AV 1/13 Serial Number: WD-WCC4MLULA4S4 1/14 Serial Number: WD-WMC6N0F8FU0U 1/15 Serial Number: WD-WCAVY0090692 1/16 Serial Number: WD-WCC6N5CLHYSS | Bay | ID | Drive | Name | Serial Number | | ---- | ---- | ----- | --------- | --------------- | | 20 | 1/1 | da0 | JBOD-1-1 | WD-WMAZA9378538 | | 21 | 1/2 | da1 | JBOD-1-2 | V1KVXSXG | | 22 | 1/3 | da2 | JBOD-1-3 | WD-WCC300755734 | | 23 | 1/4 | da3 | JBOD-1-4 | WD-WMAZA9520754 | | 00 | 1/5 | da4 | JBOD-1-5 | WD-WCAVY0114154 | | 01 | 1/6 | da5 | JBOD-1-6 | WD-WMC5C0D9L6HV | | 02 | 1/7 | da6 | JBOD-1-7 | WD-WCC6N5UY8H5P | | 03 | 1/8 | da7 | JBOD-1-8 | WD-WCC300759632 | | 30 | 1/9 | da8 | JBOD-1-9 | WD-WCC6N6DDJ7DT | | 31 | 1/10 | da9 | JBOD-1/10 | WD-WCC6N6VTE2NF | | 32 | 1/11 | da10 | JBOD-1/11 | WD-WMC5C0D8Z7FV | | 33 | 1/12 | da11 | JBOD-1/12 | WD-WMC6N0J6S9AV | | 10 | 1/13 | da12 | JBOD-1/13 | WD-WCC4MLULA4S4 | | 11 | 1/14 | da13 | JBOD-1/14 | WD-WMC6N0F8FU0U | | 12 | 1/15 | da14 | JBOD-1/15 | WD-WCAVY0090692 | | 13 | 1/16 | da15 | JBOD-1/16 | WD-WCC6N5CLHYSS | * da1 MK0171YFKMA4AG replaced 2022.05.08 * da12 MK0171YFKMA59G replaced 2022.04.12 * da8 WD-WCAVY0111166 replaced 2021.11.22 * da10 WD-WCAVY0127426 replaced 2016.09.15 * da1 S2F06GX5 replaced 2016.01.2? * da2 WD-WCAVY0112831 replaced 2014.05.12 * da5 WD-WCAVY0107135 replaced 2017.03.25 * da13 W1E0B4NJ replaced 2017.06.xx * da6 WD-WCAVY0127865 replaced 2018.01.07 * da6 WD-WMC6N0H8K48M replaced 2021.01.16 * da9 WD-WCAVY0117958 replaced 2021.03.15 * da15 WD-WCC4M5VJ4UK3 replaced 2021.06.18 * da9 WD-WMAY04440674 replaced 2021.08.27 ## raid1.dfw.rg.net Dell R530
Service Tag: HQWPW52
PERC H730P RAID Controller, 2G B NV Cache, Mini card
2 x 4TB in md0 RAID1 as opsys
5 x 16TB in zfs
### raid1 Drives `smartctl -a -d megaraid,[0-7] /dev/bus/0` | | | | | | | Bay | Manufacturer | Model | Serial | Installed | | --- | --------------- | -------------------- | -------- | ----------- | | 0 | Seagate | ST4000NM0005 | Z4C056TC | 2022.06.06 | | 1 | Seagate | ST4000NM0005 | Z4C056KT | 2022.06.06 | | 2 | Western Digital | WD161KRYZ-01AGBB0 | 2PH9PMWT | 2022.06.06 | | 3 | Western Digital | WD161KRYZ-01AGBB0 | 2PHA0RMT | 2022.06.06 | | 4 | Western Digital | WD161KRYZ-01AGBB0 | 2PH9NJNT | 2022.06.06 | | 5 | Western Digital | WD161KRYZ-01AGBB0 | 2PH69HJJ | 2022.06.06 | | 6 | Seagate | ST16000NM000J-2TW103 | ZR706BX0 | 2022.06.06 | ## DEAD dfw0.psg.com dfw0 drives `for i in 4 5 6 7; do echo /dev/ad$i; smartctl -a /dev/ad$i | egrep 'Model|Serial'; done` | | | | | | ---- | ------------------ | ------------ | ---- | | Bay | Model | Serial | | | | | | | | ada0 | ST31000340NS | 9QJ3Q29E | 1TB | | ada1 | WD20EADS-00R6B0 | WCAVY0388932 | 2TB | | ada2 | WD2005FBYZ-01YCBB1 | WMC6N0F2KZU6 | 2TB | | ada3 | ST31000340NS | 9QJ3M5N4 | 1TB | | | | | | dfw0 disk drive physical order | ada4 | ada6 | ada5 | ada7 | | ---- | ---- | ---- | ---- | | ada0 | ada2 | ada1 | ada3 | logical drive labels | label/disk00 | ada0s3 | | ------------ | ------ | | label/disk01 | ada1s3 | | label/disk02 | ada2s1 | | label/disk03 | ada3s1 | ## DEAD nfs0.dfw.rg.net ### MoBo * [PDSME+](http://www.supermicro.com/products/motherboard/Xeon3000/3010/PDSME_.cfm) * [​AOC-SIM1U+ IPMI Card](http://www.supermicro.com/products/accessories/addon/SIM.cfm) ### DEAD nfs0.dfw.rg.net Disk Drives ``` for i in 0 1 2 3 4 5 6 7; do tw_cli /c2/p$i show serial; done ``` | Port | Model | Serial | Size | Date | | -------- | ------------------ | --------------- | ---- | ------------------ | | p0 | ST4000DM000-1F2168 | Z300L98C | 4TB | don't know | | p1 | WD4000F9YZ-09N20L1 | WD-WCC5D1DS4012 | 4TB | 2019.04.07 | | p2 | WD60EFRX-68L0BN1 | WD-WX41DA57H7UJ | 6TB | 2018.03.09 | | p3 | WD4000F9YZ-09N20L1 | WD-WCC5D2KHHYE8 | 4TB | 2016.06.09 | | p4 | ST4000DM000-1F2168 | Z300KBMM | 4TB | don't know | | p5 | ST4000DM000-1F2168 | Z300KBQN | 4TB | don't know | | p6 | ST4000DM000-1F2168 | Z300L8J5 | 4TB | don't know | | p7 | WD4000F9YZ-09N20L1 | WD-WMC5D0D7V0L9 | 4TB | 2015.12.13 | | | | | | | | p1 | ST4000DM000-1F2168 | Z300FKES | 4TB | removed 2019.04.07 | | p2 | ST4000DM000-1F2168 | Z300L9T3 | 4TB | removed 2018.03.09 | | incoming | HUS724040ALS640 | PBK4K70X | 4TB | 2015.12.18 | ### Physical Drive layout | | | | | | ---- | ---- | ---- | ---- | | 4 | 5 | 6 | 7 | | 0 | 1 | 2 | 3 | ## Disgusting Details ``` r0 cisco 12406 r1 Juniper MX240 sw0 cisco 2960G-48TC-L with fiber to 12406 sw1 Juniper EX2200-48T-4G dfw0 1ru compute server SuperMicro SYS-6015W-NTRB w/ X7DWU dfw1 3ru big raid server SuperMicro CSE-836TQ-R800B w/ X7SBE dfw2 2ru Junosphere virtualization server o0 cisco 2511 with copper to sw0 and serial cables to all craft ports o1 cisco 2511 with copper to sw0 and serial cables to all craft ports o2 Cyclades® ACS Advanced Console Server pwr0 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20 pwr1 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20 --- Power SO1-A:5C2.2-62/64 SO2-A:5C2.2-66/68 SO1-B:5D2.2-62/64 SO2-B:5D2.2-66/68 C-Side is Blue and that = "A" feed D-Side is Yellow and that = "B" feed [ SO2-B:5D2.2-66/68 will not be used ] The demarc for the xconnects is as follows: pp01.dfw06 137-138 M3 Future 139-140 M4 Future 141-142 M5 Equinix IX Cable ID #20466027 Terminated on CP:0104:1068601 ports 3/4 143-144 M6 NTT/Verio r08.dllstx09.us.bb g0/6/0/29 Rge Equinix order info for the exchange fiber: Ordering contact- Chris Morrow and/or CSM- Shazyra Irizarry Technical contact – Randy Bush IBX- DA1 Order: 1-97362053795 WOA- 3-97445043133 (IX Port Install) WOA- 3-97448754941 (NCC Cable install) Serial # 20466027 ; terminated on CP:0104:1068601 ports 3/4 --- Dallas Equinix Colo, Downstairs Google demarc for the riser cable on the Equinix end is Suite 1034 Cage 00372 Cabinet 105 NTT/Verio TDI Cage 220, Cab 101 FP#1, Port NA, SMF/SC. r08.dllstx09.us.bb g0/6/0/29 Media: MMF Connector: SC --- ntt circuit --- EQUINIX INC. - Cross Connect Order Complete - 817885.1.1.2 Completed On: 08-JUN-2009 IBX: DA1 Type of Order: Cross Connect Cage: DA1::00080:EQUINIX.COM Cabinet Number: 0101 A-Side Patch Panel/Module/Port: PP:0105:2288||1 Z-Side Customer and Cage Number: YOUTUBE, LLC DA1::00372:YOUTUBE Z-Side Cabinet Number: 0105 Carrier Circuit ID(if applicable): EQUINIX SANDBOX Z-Side Patch Panel/Module/Port: VP:0105:69862||143 Cable ID: 1234912 EQUINIX INC. - Cross Connect Order Complete - 817885.2.1.2 Completed On: 05-JUN-2009 IBX: DA1 Type of Order: Cross Connect Cage: DA1::00080:EQUINIX.COM Cabinet Number: 0101 A-Side Patch Panel/Module/Port: PP:0105:74459||3 Z-Side Customer and Cage Number: NTT AMERICA, INC. DA1::00220:NTT AMERICA, INC. (Verio) Z-Side Cabinet Number: 0101 Carrier Circuit ID(if applicable): EQUINIX SANDBOX Z-Side Patch Panel/Module/Port: PP:0101:2375||15 Cable ID: 1234913 For both crossconnects, even though the "Cabinet Number" field shows "0101", the actual Equinix A-Side cab is "0105" as listed in the panel info. And to reduce cabling and points of failure, the IBX site technicians said that they would be connecting both crossconnects directly into each other rather than with an intermediate patch cable. --- equinix exchange --- csw7.da1, gig 0/35, Sandbox/RGNet, IX, circuit id 1236454 ``` ## Routers [![120131.dallas-routers.jpg](../pics/120131.dallas-routers.jpg)] ## Tapping BGP Feeds [![110617.dfw-data-tap.jpg](../pics/110617.dfw-data-tap.jpg)] ## Old Google PoP We were in Google POP in Via West ``` Viawest/Dataside 1950 Stemmons Freeway, Suite 2033 Dallas TX 75027 +1-214-231-0162 Mark for "inside delivery" Google PoP DFW06 Rack Location: 6.3 Rack: ESW E-00589-01, Black Caster and Mounting Rails, No Door, 7Ft. ``` --- 2022.06.06