382 lines
No EOL
20 KiB
Markdown
382 lines
No EOL
20 KiB
Markdown
# Dallas Google PoP
|
||
|
||
### New Google POP
|
||
|
||
```
|
||
Site DFW 28
|
||
Cage 4000.40
|
||
1950 N Stemmons Freeway
|
||
Suite 4000
|
||
Dallas, TX 75207
|
||
|
||
Racks 4-11 4-12
|
||
RGnet in 4-12
|
||
IETF/Deadalus/RGnet in 4-11
|
||
|
||
Shipping Address:
|
||
Google - chris morrow
|
||
c/o: Infomart Suite 4000
|
||
1950 N Stemmons Freeway
|
||
Dallas, TX 75207
|
||
```
|
||
|
||
|
||
|
||
## Rack Physical and Wiring
|
||
|
||
| hostname | switch | ipmi | oob | oobname | pwr | rack | |
|
||
| :--------------------------- | ------------------------------------------------ | ------------ | ------- | ---------- | ------------------------------------------------ | ----- | ------------------------------------- |
|
||
| o2.dfw.rg.net | sw99-42 | | 1.8 | o2 | | | Cyclades® ACS Advanced Console Server |
|
||
| r0.dfw.rg.net-rp0 | 0.45 | | 0.01 | r00 | subfl | 03-13 | |
|
||
| r0.dfw.rg.net-rp0mgt | 0.33 | | | | | 03-13 | |
|
||
| r0.dfw.rg.net-rp1 | 0.?? | | 0.02 | r01 | | 03-13 | |
|
||
| r0.dfw.rg.net-rp1mgt | 0.34 | | | | | 03-13 | |
|
||
| [r0.2/0/4 LAN] | 0.04 | | | | | 03-13 | |
|
||
| [r0.2/0/5 ntt fiber] | 0.05 | | | | | 03-13 | |
|
||
| [r0.2/0/6 spare] | 0.06 | | | | | 03-13 | |
|
||
| [r0.2/0/7 equinix ix] | 0.15 | | | | | 03-13 | |
|
||
| sw0.dfw.rg.net | | | 0.13 | sw0 | [^1] | 17 | |
|
||
| [equnix IX fiber] | 0.46 | | | | | | |
|
||
| r1.dfw.rg.net.2/0 | 0.07 | | | r1 | | 18-21 | |
|
||
| r1.dfw.rg.net.2/1 | 0.08 | | | | | 18-21 | |
|
||
| r1.dfw.rg.net.2/2 | 0.09 | | | | | 18-21 | |
|
||
| r1.dfw.rg.net.2/3 | 0.10 | | | | | 18-21 | |
|
||
| r1.dfw.rg.net.mgt | 0.43 | | | | | 18-21 | |
|
||
| o1.dfw.rg.net | 0.40 | | 0.aux | | 1.1 | 22 | |
|
||
| panduit cover-up | | | | | | 23 | |
|
||
| o0.dfw.rg.net | 0.44 | | 1.aux | | 0.1 | 24 | |
|
||
| pwr1.dfw.rg.net | 0.42 | o2.9 / o2.10 | 0.15 | pwr1 | subfl | 26 | |
|
||
| pwr0-blue.dfw.rg.net | 0.18 | | o2.16 | pwr0-blue | subfl | - | |
|
||
| raid0.dfw.rg.net | 0.12 | 0.26 | 0.04 | nfs0 | 0.5/1.5 | 31-33 | |
|
||
| dfw0.psg.com | 0.11 | 0.31 | 0.03 | dfw0 | 0.3/1.3 | 34 | |
|
||
| sw1.dfw.rg.net | 0.46 | | 1.01 | sw1 | [^2] | 35 | ??? |
|
||
| vm0.dfw.rg.net | [^1] | 1.22 | 1.10 | vm0 | 0.4/1.4 | 35-36 | |
|
||
| vm1.dfw.rg.net | [^1] | 1.23 | o2.41 | vm1 | 0.4/1.4 | 35-36 | |
|
||
| vm2.dfw.rg.net | [^1] | 1.24 | o2.42 | vm2 | 0.4/1.4 | 35-36 | |
|
||
| vm3.dfw.rg.net | [^1] | 1.25 | o2.43 | vm3 | 0.4/1.4 | 35-36 | |
|
||
| kingsx.richosman.com | 0.21 | | o2.48 | kingsx | 0.6/1.6 | 37 | |
|
||
| b0.noc.ietf.org | | xx | o2.45 | b0 | xx | 99 | |
|
||
| paddington.richosman | 0.22 | | o2.47 | paddington | 0.8/1.8 | 37 | |
|
||
| nfs0.dfw.rg.net | 0.12 | 0.24 | 0.27&29 | nfs0 | red 0.5/1.5 | 42-43 | |
|
||
| dallas01.mux.peering.usc.edu | 0.37/0.39 | | o2.46 | usc | 0.7/1.7 | 44 | |
|
||
|
||
|
||
|
||
## vm0/1/2/3.dfw.rg.net Ganeti Cluster / 198.180.152.30/31/32/33
|
||
|
||
VMs are [Supermicro X8DTT-HF Motherboards](http://www.supermicro.com/products/motherboard/QPI/5500/X8DTT-H.cfm?INF=) with 48G of RAM each
|
||
in [827H-R1200B Chassis](http://www.supermicro.com/products/chassis/2U/827/SC827H-R1200.cfm)
|
||
or [SC827H-R1400B Chassis](http://www.supermicro.com/products/chassis/2U/827/SC827H-R1400.cfm)
|
||
12 SATA-3 250G Drives model WDC WD2502ABYS-0.
|
||
|
||
[^1]: vm servers' ethers are connected to sw2.dfw as follows:
|
||
|
||
| vm | ipmi | e0 | e1 |
|
||
| ---- | ------ | ------ | ------ |
|
||
| vm0 | sw0-13 | sw0-19 | sw2-43 |
|
||
| vm1 | sw0-14 | sw0-20 | sw2-44 |
|
||
| vm2 | sw0-15 | sw0-21 | sw2-45 |
|
||
| vm3 | sw0-16 | sw0-22 | sw2-45 |
|
||
|
||
[^2]: PTSS/B is connected to power 0.2/1.2 and controls both sw0 and sw1
|
||
|
||
### VM Node Layout (from rear, rack DFW28-04-12)
|
||
|
||
| | |
|
||
| ---- | ---- |
|
||
| vm0 | vm2 |
|
||
| vm1 | vm3 |
|
||
| | |
|
||
|
||
|
||
|
||
### vm[0-3].dfw.rg.net Disk Drives
|
||
|
||
| Bay | vm | Model | Serial | Size | Date | |
|
||
| ---- | ---- | ------------------ | ------------ | ---- | ---------- | ----------------------------- |
|
||
| A0 | vm3 | WD2000F9YZ | WMC5C0D3HDAV | 2TB | 2015.03.16 | |
|
||
| A1 | vm3 | WD2000F9YZ | WMC5C0D3TV0R | 2TB | 2015.03.16 | |
|
||
| A2 | vm3 | WD2000F9YZ-09N20L1 | WX11D363TV6F | 6TB | 2018.01.10 | was WMC5C0D5KYPP |
|
||
| B0 | vm2 | WD2000F9YZ | WMC5C0D2NV6N | 2TB | 2015.03.16 | |
|
||
| B1 | vm2 | WD2005FBYZ | WMC6N0M3AEHR | 2TB | 2015.03.16 | WMC5C0D623ZS pulled 2018.11.3 |
|
||
| B2 | vm2 | WD2000F9YZ | WMC5C0D75CRL | 2TB | 2015.03.16 | |
|
||
| C0 | vm1 | WD2000F9YZ | WMC5C0D61JYX | 2TB | 2015.03.16 | |
|
||
| C1 | vm1 | WD2000F9YZ | WMC5C0DA1006 | 2TB | 2015.03.16 | |
|
||
| C2 | vm1 | WD2000F9YZ | WMC6N0H8FVYN | 2TB | 2015.03.16 | was WMC5C0D44TEU |
|
||
| D0 | vm0 | WD2000F9YZ | WMC5C0D2APXE | 2TB | 2015.03.16 | |
|
||
| D1 | vm0 | WD2000F9YZ | WMC5C0D34PT0 | 2TB | 2015.03.16 | |
|
||
| D2 | vm0 | WD2000F9YZ | WMC5C0D2HXNV | 2TB | 2015.03.16 | |
|
||
|
||
|
||
|
||
### Ganeti Cluster / 198.180.152.30/31/32/33
|
||
|
||
```
|
||
vm0.dfw.rg.net:/root# ./do-stat
|
||
Node DTotal DFree MTotal MNode MFree Pinst Sinst
|
||
vm0.dfw.rg.net 3.6T 3.3T 47.3G 33.9G 39.2G 3 2
|
||
vm1.dfw.rg.net 3.6T 3.3T 47.3G 21.4G 39.4G 3 3
|
||
vm2.dfw.rg.net 3.6T 3.5T 47.3G 24.4G 42.7G 3 1
|
||
vm3.dfw.rg.net 3.6T 3.0T 47.3G 22.3G 32.2G 4 2
|
||
```
|
||
|
||
### Hosts on vm0/1/2/3.dfw.rg.net
|
||
|
||
| Instance | Primary_node | Secondary_Node | MaxMem | DiskUsage | IP_Address | SysAdmin |
|
||
| ------------------- | -------------- | -------------- | ------ | --------- | --------------- | ------------------------------------------ |
|
||
| ca0.rpki.net | vm0.dfw.rg.net | vm2.dfw.rg.net | 2.0G | 128.1G | 198.180.152.16 | Randy Bush <randy@psg.com> |
|
||
| ca.rg.net | vm0.dfw.rg.net | vm3.dfw.rg.net | 4.0G | 128.0G | 198.180.152.6 | Randy Bush <randy@psg.com> |
|
||
| cu0.dfw.rg.net | vm2.dfw.rg.net | vm3.dfw.rg.net | 512M | 12.0G | 198.180.152.71 | Steven M Bellovin <smb@cs.columbia.edu> John Koh <koh@…> |
|
||
| git.rg.net | vm3.dfw.rg.net | vm2.dfw.rg.net | 2.0G | 34.0G | 198.180.152.8 | Randy Bush <randy@psg.com> |
|
||
| icir.dfw.rg.net | vm3.dfw.rg.net | vm2.dfw.rg.net | 1.0G | 32.0G | 198.180.152.57 | Bill Marczak <billmar@gmail.com> |
|
||
| netsrv.dfw.rg.net | vm3.dfw.rg.net | | 1.0G | 100G | 198.180.152.15 | Randy Bush <randy@psg.com> |
|
||
| nms.rg.net | vm3.dfw.rg.net | vm0.dfw.rg.net | 2.0G | 20G | 198.180.152.9 | Randy Bush <randy@psg.com> |
|
||
| petunia.hactrn.net | vm2.dfw.rg,net | | 6G | 120G | 198.180.152.200 | Rob Austein <sra@hactrn.net> |
|
||
| pfs0.dfw.rg.net | vm0.dfw.rg.net | vm2.dfw.rg.net | 512M | 8.1G | 198.180.152.62 | Randy Bush <randy@psg.com> |
|
||
| rb-build.dfw.rg.net | vm2.dfw.rg.net | vm1.dfw.rg.net | 16G | 150.1G | 198.180.152.62 | Randy Bush <randy@psg.com> |
|
||
| ymbk.com | vm2.dfw.rg.net | vm3.dfw.rg.net | 512M | 16.1G | 198.180.152.10 | Randy Bush <randy@psg.com> |
|
||
|
||
|
||
|
||
|
||
### nfs0.dfw.rg.net
|
||
|
||
[PDSME+](http://www.supermicro.com/products/motherboard/Xeon3000/3010/PDSME_.cfm)
|
||
[AOC-SIM1U+ IPMI Card](http://www.supermicro.com/products/accessories/addon/SIM.cfm)
|
||
|
||
### nfs0.dfw.rg.net Disk Drives
|
||
|
||
```
|
||
for i in 0 1 2 3 4 5 6 7; do tw_cli /c2/p$i show serial; done
|
||
```
|
||
|
||
| Port | Model | Serial | Size | Date |
|
||
| -------- | ------------------ | --------------- | ---- | ------------------ |
|
||
| p0 | ST4000DM000-1F2168 | Z300L98C | 4TB | don't know |
|
||
| p1 | WD4000F9YZ-09N20L1 | WD-WCC5D1DS4012 | 4TB | 2019.04.07 |
|
||
| p2 | WD60EFRX-68L0BN1 | WD-WX41DA57H7UJ | 6TB | 2018.03.09 |
|
||
| p3 | WD4000F9YZ-09N20L1 | WD-WCC5D2KHHYE8 | 4TB | 2016.06.09 |
|
||
| p4 | ST4000DM000-1F2168 | Z300KBMM | 4TB | don't know |
|
||
| p5 | ST4000DM000-1F2168 | Z300KBQN | 4TB | don't know |
|
||
| p6 | ST4000DM000-1F2168 | Z300L8J5 | 4TB | don't know |
|
||
| p7 | WD4000F9YZ-09N20L1 | WD-WMC5D0D7V0L9 | 4TB | 2015.12.13 |
|
||
| | | | | |
|
||
| p1 | ST4000DM000-1F2168 | Z300FKES | 4TB | removed 2019.04.07 |
|
||
| p2 | ST4000DM000-1F2168 | Z300L9T3 | 4TB | removed 2018.03.09 |
|
||
| incoming | HUS724040ALS640 | PBK4K70X | 4TB | 2015.12.18 |
|
||
|
||
|
||
Physical layout is
|
||
|
||
| 4 | 5 | 6 | 7 |
|
||
| ---- | ---- | ---- | ---- |
|
||
| 0 | 1 | 2 | 3 |
|
||
|
||
### raid0.dfw.rg.net
|
||
|
||
[Mobo Supermicro X7SB4/E](https://www.supermicro.com/products/motherboard/Xeon3000/3210/X7SB4.cfm) [IPMI AOC-IPMI20-E](http://www.supermicro.com/products/accessories/addon/AOC-IPMI20-E.cfm)
|
||
|
||
raid0 disk drive physical order markings on drive carriers
|
||
|
||
| 00 | 10 | 20 | 30 |
|
||
| ---- | ---- | ---- | ---- |
|
||
| 01 | 11 | 21 | 31 |
|
||
| 02 | 12 | 22 | 32 |
|
||
| 03 | 13 | 23 | 33 |
|
||
|
||
```
|
||
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do
|
||
echo -n "1/$i ";
|
||
smartctl -i -d hpt,1/$i /dev/hptrr | grep ^Serial;
|
||
done
|
||
```
|
||
|
||
| Bay | ID | Drive | Name | Serial Number |
|
||
| ---- | ---- | ----- | --------- | --------------- |
|
||
| 20 | 1/1 | da0 | JBOD-1-1 | WD-WMAZA9378538 |
|
||
| 21 | 1/2 | da1 | JBOD-1-2 | MK0171YFKMA4AG |
|
||
| 22 | 1/3 | da2 | JBOD-1-3 | WD-WCC300755734 |
|
||
| 23 | 1/4 | da3 | JBOD-1-4 | WD-WMAZA9520754 |
|
||
| 00 | 1/5 | da4 | JBOD-1-5 | WD-WCAVY0114154 |
|
||
| 01 | 1/6 | da5 | JBOD-1-6 | WD-WMC5C0D9L6HV |
|
||
| 02 | 1/7 | da6 | JBOD-1-7 | WD-WMC6N0H8K48M |
|
||
| 03 | 1/8 | da7 | JBOD-1-8 | WD-WCC300759632 |
|
||
| 30 | 1/9 | da8 | JBOD-1-9 | WD-WCAVY0111166 |
|
||
| 31 | 1/10 | da9 | JBOD-1/10 | WD-WCAVY0117958 |
|
||
| 32 | 1/11 | da10 | JBOD-1/11 | WD-WMC5C0D8Z7FV |
|
||
| 33 | 1/12 | da11 | JBOD-1/12 | WD-WMC6N0J6S9AV |
|
||
| 10 | 1/13 | da12 | JBOD-1/13 | MK0171YFKMA59G |
|
||
| 11 | 1/14 | da13 | JBOD-1/14 | WD-WMC6N0F8FU0U |
|
||
| 12 | 1/15 | da14 | JBOD-1/15 | WD-WCAVY0090692 |
|
||
| 13 | 1/16 | da15 | JBOD-1/16 | WD-WCC4M5VJ4UK3 |
|
||
|
||
da10 WD-WCAVY0127426 replaced 2016.09.15
|
||
da1 S2F06GX5 replaced 2016.01.2
|
||
da2 WD-WCAVY0112831 replaced 2014.05.12
|
||
da5 WD-WCAVY0107135 replaced 2017.03.25
|
||
da13 W1E0B4NJ replaced 2017.06.xx
|
||
da6 WD-WCAVY0127865 replaced 2018.01.07
|
||
|
||
Spare Drives
|
||
|
||
| | | |
|
||
| ---- | ------------- | ------------ |
|
||
| Size | Serial Number | Model # |
|
||
| 2tb | WMC6NOF8FU0U | WD2005FBYZSP |
|
||
| 2th | WMC6NOF2KZU6 | WD2005FBYZSP |
|
||
| | | |
|
||
|
||
### dfw0.psg.com
|
||
|
||
dfw0 drives `for i in 4 5 6 7; do echo /dev/ad$i; smartctl -a /dev/ad$i | egrep 'Model|Serial'; done`
|
||
|
||
| | | | |
|
||
| ---- | ------------------ | ------------ | ---- |
|
||
| Bay | Model | Serial | |
|
||
| | | | |
|
||
| ada0 | ST31000340NS | 9QJ3Q29E | 1TB |
|
||
| ada1 | WD20EADS-00R6B0 | WCAVY0388932 | 2TB |
|
||
| ada2 | WD2005FBYZ-01YCBB1 | WMC6N0F2KZU6 | 2TB |
|
||
| ada3 | ST31000340NS | 9QJ3M5N4 | 1TB |
|
||
| | | | |
|
||
|
||
dfw0 disk drive physical order
|
||
|
||
| ada4 | ada6 | ada5 | ada7 |
|
||
| ---- | ---- | ---- | ---- |
|
||
| ada0 | ada2 | ada1 | ada3 |
|
||
|
||
logical drive labels
|
||
|
||
| label/disk00 | ada0s3 |
|
||
| ------------ | ------ |
|
||
| label/disk01 | ada1s3 |
|
||
| label/disk02 | ada2s1 |
|
||
| label/disk03 | ada3s1 |
|
||
|
||
## Disgusting Details
|
||
|
||
```
|
||
r0 cisco 12406
|
||
r1 Juniper MX240
|
||
sw0 cisco 2960G-48TC-L with fiber to 12406
|
||
sw1 Juniper EX2200-48T-4G
|
||
dfw0 1ru compute server SuperMicro SYS-6015W-NTRB w/ X7DWU
|
||
dfw1 3ru big raid server SuperMicro CSE-836TQ-R800B w/ X7SBE
|
||
dfw2 2ru Junosphere virtualization server
|
||
o0 cisco 2511 with copper to sw0 and serial cables to all craft ports
|
||
o1 cisco 2511 with copper to sw0 and serial cables to all craft ports
|
||
o2 Cyclades® ACS Advanced Console Server
|
||
pwr0 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20
|
||
pwr1 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20
|
||
|
||
---
|
||
|
||
Power
|
||
SO1-A:5C2.2-62/64
|
||
SO2-A:5C2.2-66/68
|
||
SO1-B:5D2.2-62/64
|
||
SO2-B:5D2.2-66/68
|
||
C-Side is Blue and that = "A" feed
|
||
D-Side is Yellow and that = "B" feed
|
||
[ SO2-B:5D2.2-66/68 will not be used ]
|
||
|
||
The demarc for the xconnects is as follows:
|
||
pp01.dfw06
|
||
137-138 M3 Future
|
||
139-140 M4 Future
|
||
141-142 M5 Equinix IX Cable ID #20466027 Terminated on CP:0104:1068601 ports 3/4
|
||
143-144 M6 NTT/Verio r08.dllstx09.us.bb g0/6/0/29
|
||
|
||
Rge Equinix order info for the exchange fiber:
|
||
Ordering contact- Chris Morrow and/or CSM- Shazyra Irizarry
|
||
Technical contact – Randy Bush <randy@psg.com>
|
||
IBX- DA1
|
||
Order: 1-97362053795
|
||
WOA- 3-97445043133 (IX Port Install)
|
||
WOA- 3-97448754941 (NCC Cable install)
|
||
Serial # 20466027 ; terminated on CP:0104:1068601 ports 3/4
|
||
|
||
---
|
||
|
||
Dallas Equinix Colo, Downstairs
|
||
|
||
Google demarc for the riser cable on the Equinix end is Suite 1034 Cage
|
||
00372 Cabinet 105
|
||
|
||
NTT/Verio TDI
|
||
Cage 220, Cab 101 FP#1, Port NA, SMF/SC.
|
||
r08.dllstx09.us.bb g0/6/0/29
|
||
|
||
Media: MMF
|
||
Connector: SC
|
||
|
||
--- ntt circuit ---
|
||
|
||
EQUINIX INC. - Cross Connect Order Complete - 817885.1.1.2
|
||
Completed On: 08-JUN-2009
|
||
IBX: DA1
|
||
Type of Order: Cross Connect
|
||
Cage: DA1::00080:EQUINIX.COM
|
||
Cabinet Number: 0101
|
||
A-Side Patch Panel/Module/Port: PP:0105:2288||1
|
||
Z-Side Customer and Cage Number: YOUTUBE, LLC
|
||
DA1::00372:YOUTUBE
|
||
Z-Side Cabinet Number: 0105
|
||
Carrier Circuit ID(if applicable): EQUINIX SANDBOX
|
||
Z-Side Patch Panel/Module/Port: VP:0105:69862||143
|
||
Cable ID: 1234912
|
||
|
||
EQUINIX INC. - Cross Connect Order Complete - 817885.2.1.2
|
||
Completed On: 05-JUN-2009
|
||
IBX: DA1
|
||
Type of Order: Cross Connect
|
||
Cage: DA1::00080:EQUINIX.COM
|
||
Cabinet Number: 0101
|
||
A-Side Patch Panel/Module/Port: PP:0105:74459||3
|
||
Z-Side Customer and Cage Number: NTT AMERICA, INC.
|
||
DA1::00220:NTT AMERICA, INC. (Verio)
|
||
Z-Side Cabinet Number: 0101
|
||
Carrier Circuit ID(if applicable): EQUINIX SANDBOX
|
||
Z-Side Patch Panel/Module/Port: PP:0101:2375||15
|
||
Cable ID: 1234913
|
||
|
||
For both crossconnects, even though the "Cabinet Number" field shows
|
||
"0101", the actual Equinix A-Side cab is "0105" as listed in the panel
|
||
info. And to reduce cabling and points of failure, the IBX site
|
||
technicians said that they would be connecting both crossconnects
|
||
directly into each other rather than with an intermediate patch cable.
|
||
|
||
--- equinix exchange ---
|
||
|
||
csw7.da1, gig 0/35, Sandbox/RGNet, IX, circuit id 1236454
|
||
```
|
||
|
||
## Routers
|
||
|
||
[]
|
||
|
||
## Tapping BGP Feeds
|
||
|
||
[]
|
||
|
||
|
||
## Old Google PoP
|
||
|
||
We were in Google POP in Via West
|
||
|
||
```
|
||
Viawest/Dataside
|
||
1950 Stemmons Freeway, Suite 2033
|
||
Dallas TX 75027
|
||
+1-214-231-0162
|
||
Mark for "inside delivery"
|
||
|
||
Google PoP DFW06
|
||
Rack Location: 6.3
|
||
Rack: ESW E-00589-01, Black Caster and Mounting Rails, No Door, 7Ft.
|
||
```
|
||
|
||
---
|
||
|
||
2020.11.07 |