# Dallas Google PoP

## New Google POP

```
Site DFW 28
Cage 4000.40
1950 N Stemmons Freeway
Suite 4000
Dallas, TX 75207

Racks 4-11 4-12
  RGnet in 4-12
  IETF/Deadalus/RGnet in 4-11

Shipping Address:
 Google - chris morrow
 c/o: Infomart Suite 4000
 1950 N Stemmons Freeway
 Dallas, TX 75207
```



## Rack Physical and Wiring

| hostname                     | switch                                           | ipmi         | oob     | oobname    | pwr                                              | rack  |                                       |
| :--------------------------- | ------------------------------------------------ | ------------ | ------- | ---------- | ------------------------------------------------ | ----- | ------------------------------------- |
| o2.dfw.rg.net                | sw99-42                                          |              | 1.8     | o2         |                                                  |       | Cyclades® ACS Advanced Console Server |
| r0.dfw.rg.net-rp0            | 0.45                                             |              | 0.01    | r00        | subfl                                            | 03-13 |                                       |
| r0.dfw.rg.net-rp0mgt         | 0.33                                             |              |         |            |                                                  | 03-13 |                                       |
| r0.dfw.rg.net-rp1            | 0.??                                             |              | 0.02    | r01        |                                                  | 03-13 |                                       |
| r0.dfw.rg.net-rp1mgt         | 0.34                                             |              |         |            |                                                  | 03-13 |                                       |
| [r0.2/0/4 LAN]               | 0.04                                             |              |         |            |                                                  | 03-13 |                                       |
| [r0.2/0/5 ntt fiber]         | 0.05                                             |              |         |            |                                                  | 03-13 |                                       |
| [r0.2/0/6 spare]             | 0.06                                             |              |         |            |                                                  | 03-13 |                                       |
| [r0.2/0/7 equinix ix]        | 0.15                                             |              |         |            |                                                  | 03-13 |                                       |
| sw0.dfw.rg.net               |                                                  |              | 0.13    | sw0        | [^1] | 17    |                                       |
| [equnix IX fiber]            | 0.46                                             |              |         |            |                                                  |       |                                       |
| r1.dfw.rg.net.2/0            | 0.07                                             |              |         | r1         |                                                  | 18-21 |                                       |
| r1.dfw.rg.net.2/1            | 0.08                                             |              |         |            |                                                  | 18-21 |                                       |
| r1.dfw.rg.net.2/2            | 0.09                                             |              |         |            |                                                  | 18-21 |                                       |
| r1.dfw.rg.net.2/3            | 0.10                                             |              |         |            |                                                  | 18-21 |                                       |
| r1.dfw.rg.net.mgt            | 0.43                                             |              |         |            |                                                  | 18-21 |                                       |
| o1.dfw.rg.net                | 0.40                                             |              | 0.aux   |            | 1.1                                              | 22    |                                       |
| panduit cover-up             |                                                  |              |         |            |                                                  | 23    |                                       |
| o0.dfw.rg.net                | 0.44                                             |              | 1.aux   |            | 0.1                                              | 24    |                                       |
| pwr1.dfw.rg.net              | 0.42                                             | o2.9 / o2.10 | 0.15    | pwr1       | subfl                                            | 26    |                                       |
| pwr0-blue.dfw.rg.net         | 0.18                                             |              | o2.16   | pwr0-blue  | subfl                                            | -     |                                       |
| raid0.dfw.rg.net             | 0.12                                             | 0.26         | 0.04    | nfs0       | 0.5/1.5                                          | 31-33 |                                       |
| dfw0.psg.com                 | 0.11                                             | 0.31         | 0.03    | dfw0       | 0.3/1.3                                          | 34    |                                       |
| sw1.dfw.rg.net               | 0.46                                             |              | 1.01    | sw1        | [^2] | 35    | ???                                   |
| vm0.dfw.rg.net               | [^1] | 1.22         | 1.10    | vm0        | 0.4/1.4                                          | 35-36 |                                       |
| vm1.dfw.rg.net               | [^1] | 1.23         | o2.41   | vm1        | 0.4/1.4                                          | 35-36 |                                       |
| vm2.dfw.rg.net               | [^1] | 1.24         | o2.42   | vm2        | 0.4/1.4                                          | 35-36 |                                       |
| vm3.dfw.rg.net               | [^1] | 1.25         | o2.43   | vm3        | 0.4/1.4                                          | 35-36 |                                       |
| kingsx.richosman.com         | 0.21                                             |              | o2.48   | kingsx     | 0.6/1.6                                          | 37    |                                       |
| b0.noc.ietf.org              |                                                  | xx           | o2.45   | b0         | xx                                               | 99    |                                       |
| paddington.richosman         | 0.22                                             |              | o2.47   | paddington | 0.8/1.8                                          | 37    |                                       |
| nfs0.dfw.rg.net              | 0.12                                             | 0.24         | 0.27&29 | nfs0       | pwr0r 0.5/1.5                                    | 42-43 |                                       |
| dallas01.mux.peering.usc.edu | 0.37/0.39                                        |              | o2.46   | usc        | 0.7/1.7                                          | 44    |                                       |



## vm0/1/2/3.dfw.rg.net Ganeti Cluster / 198.180.152.30/31/32/33

VMs are [Supermicro X8DTT-HF Motherboards](http://www.supermicro.com/products/motherboard/QPI/5500/X8DTT-H.cfm?INF=) with 48G of RAM each
 in [​827H-R1200B Chassis](http://www.supermicro.com/products/chassis/2U/827/SC827H-R1200.cfm)
 or [​SC827H-R1400B Chassis](http://www.supermicro.com/products/chassis/2U/827/SC827H-R1400.cfm)
 12 SATA-3 250G Drives model WDC WD2502ABYS-0. 

[^1]: vm servers' ethers are connected to sw2.dfw as follows:

| vm   | ipmi   | e0     | e1     |
| ---- | ------ | ------ | ------ |
| vm0  | sw0-13 | sw0-19 | sw2-43 |
| vm1  | sw0-14 | sw0-20 | sw2-44 |
| vm2  | sw0-15 | sw0-21 | sw2-45 |
| vm3  | sw0-16 | sw0-22 | sw2-45 |

[^2]: PTSS/B is connected to power 0.2/1.2 and controls both sw0 and sw1 

### VM Node Layout (from rear, rack DFW28-04-12)

|      |      |
| ---- | ---- |
| vm0  | vm2  |
| vm1  | vm3  |
|      |      |


### vm[0-3].dfw.rg.net Disk Drives

`for i in a b c; do echo -n "sd$i "; smartctl -a /dev/sd$i | grep Serial; done`

| Bay  | vm   | drive | Model              | Serial       | Size | Date       |                               |
| ---- | ---- | ----- | ---------- | ------------ | ---- | ---------- | ----------------------------- |
| A0   | vm3  | sdc   | WD2000F9YZ | WMC5C0D3HDAV | 2TB  | 2015.03.16 |                               |
| A1   | vm3  | sdb   | WD2000F9YZ | WMC5C0D3TV0R | 2TB  | 2015.03.16 |                               |
| A2   | vm3  | sda   | WD2000F9YZ | WX11D363TV6F | 6TB  | 2018.01.10 | was WMC5C0D5KYPP |
| B0   | vm2  | sdc   | WD2000F9YZ | WMC5C0D2NV6N | 2TB  | 2015.03.16 |                               |
| B1   | vm2  | sdb   | WD2005FBYZ | WMC6N0M3AEHR | 2TB  | 2018.11.03 | was WMC5C0D623ZS |
| B2   | vm2  | sda   | WD2000F9YZ | WMC5C0D75CRL | 2TB  | 2015.03.16 |                               |
| C0   | vm1  | sdc   | WD2000F9YZ | WMC5C0D61JYX | 2TB  | 2015.03.16 |                               |
| C1   | vm1  | sdb   | WD2000F9YZ | WMC5C0DA1006 | 2TB  | 2015.03.16 |                               |
| C2   | vm1  | sda   | WD2000F9YZ | WMC6N0H8FVYN | 2TB  | 2018.01.07 | was WMC5C0D44TEU |
| D0   | vm0  | sdc   | WD2003FYYS | WMAY00749145 | 2TB  | 2021.02.21 | was WMC5C0D2APXE |
| D1   | vm0  | sdb   | WD2000F9YZ | WMC5C0D34PT0 | 2TB  | 2015.03.16 |                               |
| D2   | vm0  | sda   | WD2000F9YZ | WMC5C0D2HXNV | 2TB  | 2015.03.16 |                               |


### Ganeti Cluster / 198.180.152.30/31/32/33

```
vm0.dfw.rg.net:/root# ./do-stat 
Node           DTotal DFree MTotal MNode MFree Pinst Sinst
vm0.dfw.rg.net   3.6T  3.2T  47.2G 42.4G 39.6G     3     1
vm1.dfw.rg.net   3.6T  3.4T  47.2G  908M 45.5G     0     1
vm2.dfw.rg.net   3.6T  2.7T  47.2G 41.0G 36.1G     4     4
vm3.dfw.rg.net   3.6T  2.8T  47.2G 40.1G 37.3G     4     3
```

#### Hosts on vm0/1/2/3.dfw.rg.net

| Instance            | Primary_node   | Secondary_Node | MaxMem | DiskUsage | IP_Address      | SysAdmin                                   |
| ------------------- | -------------- | -------------- | ------ | --------- | --------------- | ------------------------------------------ |
| ca0.rpki.net        | vm0.dfw.rg.net | vm2.dfw.rg.net | 2.0G   | 128.1G    | 198.180.152.16  | Randy Bush <randy@psg.com>                       |
| ca.rg.net           | vm0.dfw.rg.net | vm3.dfw.rg.net | 4.0G   | 128.0G    | 198.180.152.6   | Randy Bush <randy@psg.com>                       |
| cu0.dfw.rg.net      | vm2.dfw.rg.net | vm3.dfw.rg.net | 512M   | 12.0G     | 198.180.152.71  | Steven M Bellovin <smb@cs.columbia.edu> John Koh <koh@…> |
| git.rg.net          | vm3.dfw.rg.net | vm2.dfw.rg.net | 2.0G   | 34.0G     | 198.180.152.8   | Randy Bush <randy@psg.com>                       |
| icir.dfw.rg.net     | vm3.dfw.rg.net | vm2.dfw.rg.net | 1.0G   | 32.0G     | 198.180.152.57  | Bill Marczak <billmar@gmail.com>                 |
| netsrv.dfw.rg.net   | vm3.dfw.rg.net |                | 1.0G   | 100G      | 198.180.152.15  | Randy Bush <randy@psg.com>                       |
| nms.rg.net          | vm3.dfw.rg.net | vm0.dfw.rg.net | 2.0G   | 20G       | 198.180.152.9   | Randy Bush <randy@psg.com>                       |
| petunia.hactrn.net  | vm2.dfw.rg,net |                | 6G     | 120G      | 198.180.152.200 | Rob Austein <sra@hactrn.net>                     |
| pfs0.dfw.rg.net     | vm0.dfw.rg.net | vm2.dfw.rg.net | 512M   | 8.1G      | 198.180.152.62  | Randy Bush <randy@psg.com>                       |
| rb-build.dfw.rg.net | vm1.dfw.rg.net | vm2.dfw.rg.net | 16G    | 150.1G    | 198.180.152.62  | Randy Bush <randy@psg.com>                       |
| ymbk.com            | vm2.dfw.rg.net | vm3.dfw.rg.net | 512M   | 16.1G     | 198.180.152.10  | Randy Bush <randy@psg.com>                       |


## nfs0.dfw.rg.net

### MoBo
* [PDSME+](http://www.supermicro.com/products/motherboard/Xeon3000/3010/PDSME_.cfm)
* [​AOC-SIM1U+ IPMI Card](http://www.supermicro.com/products/accessories/addon/SIM.cfm)

### nfs0.dfw.rg.net Disk Drives
```
for i in 0 1 2 3 4 5 6 7; do tw_cli /c2/p$i show serial; done
```

| Port     | Model              | Serial          | Size | Date               |
| -------- | ------------------ | --------------- | ---- | ------------------ |
| p0       | ST4000DM000-1F2168 | Z300L98C        | 4TB  | don't know         |
| p1       | WD4000F9YZ-09N20L1 | WD-WCC5D1DS4012 | 4TB  | 2019.04.07         |
| p2       | WD60EFRX-68L0BN1   | WD-WX41DA57H7UJ | 6TB  | 2018.03.09         |
| p3       | WD4000F9YZ-09N20L1 | WD-WCC5D2KHHYE8 | 4TB  | 2016.06.09         |
| p4       | ST4000DM000-1F2168 | Z300KBMM        | 4TB  | don't know         |
| p5       | ST4000DM000-1F2168 | Z300KBQN        | 4TB  | don't know         |
| p6       | ST4000DM000-1F2168 | Z300L8J5        | 4TB  | don't know         |
| p7       | WD4000F9YZ-09N20L1 | WD-WMC5D0D7V0L9 | 4TB  | 2015.12.13         |
|          |                    |                 |      |                    |
| p1       | ST4000DM000-1F2168 | Z300FKES        | 4TB  | removed 2019.04.07 |
| p2       | ST4000DM000-1F2168 | Z300L9T3        | 4TB  | removed 2018.03.09 |
| incoming | HUS724040ALS640    | PBK4K70X        | 4TB  | 2015.12.18         |

### Physical Drive layout

|      |      |      |      |
| ---- | ---- | ---- | ---- |
| 4    | 5    | 6    | 7    |
| 0    | 1    | 2    | 3    |

## raid0.dfw.rg.net

[Mobo Supermicro X7SB4/E](https://www.supermicro.com/products/motherboard/Xeon3000/3210/X7SB4.cfm) [IPMI AOC-IPMI20-E](http://www.supermicro.com/products/accessories/addon/AOC-IPMI20-E.cfm)

### raid0 disk drive physical order markings on drive carriers

|      |      |      |      |
| ---- | ---- | ---- | ---- |
| 00   | 10   | 20   | 30   |
| 01   | 11   | 21   | 31   |
| 02   | 12   | 22   | 32   |
| 03   | 13   | 23   | 33   |

```
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do
   echo -n "1/$i  ";
   smartctl -i -d hpt,1/$i /dev/hptrr | grep ^Serial;
   done
```

| Bay  | ID   | Drive | Name      | Serial Number   |
| ---- | ---- | ----- | --------- | --------------- |
| 20   | 1/1  | da0   | JBOD-1-1  | WD-WMAZA9378538 |
| 21   | 1/2  | da1   | JBOD-1-2  | MK0171YFKMA4AG  |
| 22   | 1/3  | da2   | JBOD-1-3  | WD-WCC300755734 |
| 23   | 1/4  | da3   | JBOD-1-4  | WD-WMAZA9520754 |
| 00   | 1/5  | da4   | JBOD-1-5  | WD-WCAVY0114154 |
| 01   | 1/6  | da5   | JBOD-1-6  | WD-WMC5C0D9L6HV |
| 02   | 1/7  | da6   | JBOD-1-7  | WD-WCC6N5UY8H5P |
| 03   | 1/8  | da7   | JBOD-1-8  | WD-WCC300759632 |
| 30   | 1/9  | da8   | JBOD-1-9  | WD-WCAVY0111166 |
| 31   | 1/10 | da9   | JBOD-1/10 | WD-WCAVY0117958 |
| 32   | 1/11 | da10  | JBOD-1/11 | WD-WMC5C0D8Z7FV |
| 33   | 1/12 | da11  | JBOD-1/12 | WD-WMC6N0J6S9AV |
| 10   | 1/13 | da12  | JBOD-1/13 | MK0171YFKMA59G  |
| 11   | 1/14 | da13  | JBOD-1/14 | WD-WMC6N0F8FU0U |
| 12   | 1/15 | da14  | JBOD-1/15 | WD-WCAVY0090692 |
| 13   | 1/16 | da15  | JBOD-1/16 | WD-WCC4M5VJ4UK3 |

* da10 WD-WCAVY0127426 replaced 2016.09.15
* da1 S2F06GX5  replaced 2016.01.2
* da2 WD-WCAVY0112831 replaced 2014.05.12
* da5 WD-WCAVY0107135 replaced 2017.03.25
* da13 W1E0B4NJ replaced 2017.06.xx
* da6 WD-WCAVY0127865 replaced 2018.01.07
* da6 WD-WMC6N0H8K48M replaced 2021.01.16

## dfw0.psg.com

dfw0 drives `for i in 4 5 6 7; do echo /dev/ad$i; smartctl -a /dev/ad$i | egrep 'Model|Serial'; done`

|      |                    |              |      |
| ---- | ------------------ | ------------ | ---- |
| Bay  | Model              | Serial       |      |
|      |                    |              |      |
| ada0 | ST31000340NS       | 9QJ3Q29E     | 1TB  |
| ada1 | WD20EADS-00R6B0    | WCAVY0388932 | 2TB  |
| ada2 | WD2005FBYZ-01YCBB1 | WMC6N0F2KZU6 | 2TB  |
| ada3 | ST31000340NS       | 9QJ3M5N4     | 1TB  |
|      |                    |              |      |

dfw0 disk drive physical order

| ada4 | ada6 | ada5 | ada7 |
| ---- | ---- | ---- | ---- |
| ada0 | ada2 | ada1 | ada3 |

logical drive labels

| label/disk00 | ada0s3 |
| ------------ | ------ |
| label/disk01 | ada1s3 |
| label/disk02 | ada2s1 |
| label/disk03 | ada3s1 |

## Disgusting Details

```
r0 cisco 12406
r1 Juniper MX240
sw0 cisco 2960G-48TC-L with fiber to 12406
sw1 Juniper EX2200-48T-4G
dfw0 1ru compute server SuperMicro SYS-6015W-NTRB w/ X7DWU
dfw1 3ru big raid server SuperMicro CSE-836TQ-R800B w/ X7SBE
dfw2 2ru Junosphere virtualization server
o0 cisco 2511 with copper to sw0 and serial cables to all craft ports 
o1 cisco 2511 with copper to sw0 and serial cables to all craft ports 
o2 Cyclades® ACS Advanced Console Server
pwr0 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20
pwr1 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20

---

Power
    SO1-A:5C2.2-62/64
    SO2-A:5C2.2-66/68
    SO1-B:5D2.2-62/64
    SO2-B:5D2.2-66/68
    C-Side is Blue and that = "A" feed
    D-Side is Yellow and that = "B" feed
    [ SO2-B:5D2.2-66/68 will not be used ]

The demarc for the xconnects is as follows:
    pp01.dfw06
    137-138 M3 Future
    139-140 M4 Future
    141-142 M5 Equinix IX Cable ID #20466027 Terminated on CP:0104:1068601 ports 3/4
    143-144 M6 NTT/Verio r08.dllstx09.us.bb g0/6/0/29

Rge Equinix order info for the exchange fiber:
Ordering contact- Chris Morrow and/or CSM- Shazyra Irizarry
Technical contact – Randy Bush  <randy@psg.com>
IBX- DA1
Order:  1-97362053795
WOA-    3-97445043133 (IX Port Install)
WOA-    3-97448754941 (NCC Cable install)
Serial # 20466027 ; terminated on CP:0104:1068601 ports 3/4

---

Dallas Equinix Colo, Downstairs

Google demarc for the riser cable on the Equinix end is Suite 1034 Cage
00372 Cabinet 105

NTT/Verio TDI
Cage 220, Cab 101 FP#1, Port NA, SMF/SC.
r08.dllstx09.us.bb g0/6/0/29

Media: MMF
Connector: SC

---  ntt circuit ---

EQUINIX INC. - Cross Connect Order Complete - 817885.1.1.2
Completed On: 08-JUN-2009
IBX: DA1
Type of Order: Cross Connect
Cage: DA1::00080:EQUINIX.COM
Cabinet Number: 0101
A-Side Patch Panel/Module/Port: PP:0105:2288||1
Z-Side Customer and Cage Number: YOUTUBE, LLC     
     DA1::00372:YOUTUBE
Z-Side Cabinet Number: 0105
Carrier Circuit ID(if applicable): EQUINIX SANDBOX
Z-Side Patch Panel/Module/Port: VP:0105:69862||143
Cable ID: 1234912

EQUINIX INC. - Cross Connect Order Complete - 817885.2.1.2
Completed On: 05-JUN-2009
IBX: DA1
Type of Order: Cross Connect
Cage: DA1::00080:EQUINIX.COM
Cabinet Number: 0101
A-Side Patch Panel/Module/Port: PP:0105:74459||3
Z-Side Customer and Cage Number: NTT AMERICA, INC.
     DA1::00220:NTT AMERICA, INC. (Verio)
Z-Side Cabinet Number: 0101
Carrier Circuit ID(if applicable): EQUINIX SANDBOX
Z-Side Patch Panel/Module/Port: PP:0101:2375||15
Cable ID: 1234913

For both crossconnects, even though the "Cabinet Number" field shows
"0101", the actual Equinix A-Side cab is "0105" as listed in the panel
info.  And to reduce cabling and points of failure, the IBX site
technicians said that they would be connecting both crossconnects
directly into each other rather than with an intermediate patch cable.

--- equinix exchange ---

csw7.da1, gig 0/35, Sandbox/RGNet, IX, circuit id 1236454
```

## Routers

[![120131.dallas-routers.jpg](../pics/120131.dallas-routers.jpg)]

## Tapping BGP Feeds

[![110617.dfw-data-tap.jpg](../pics/110617.dfw-data-tap.jpg)]


## Old Google PoP

We were in Google POP in Via West

```
Viawest/Dataside
1950 Stemmons Freeway, Suite 2033
Dallas TX 75027
+1-214-231-0162
Mark for "inside delivery"

Google PoP DFW06
Rack Location: 6.3
Rack: ESW E-00589-01, Black Caster and Mounting Rails, No Door, 7Ft.
```

---

2021.02.21