Randy Bush 6691147783 pwr0-blue replaced and outlets moved around

2025-02-01 08:24:11 -08:00

24 KiB

Raw Blame History

Dallas Google PoP

New Google POP

Site DFW 28
Cage 4000.40
1950 N Stemmons Freeway
Suite 4000
Dallas, TX 75207

Racks 4-11 4-12
  RGnet in 4-12
  IETF/Infra in 4-11

Shipping Address:
 Google - chris morrow
 c/o: Infomart Suite 4000
 1950 N Stemmons Freeway
 Dallas, TX 75207

Infra Hardware Inventory

r1 Juniper MX240
r2 cisco 9001
sw0 cisco 2960G-48TC-L with fiber to 12406
sw2 Nexus3064 48x10GE
sw3 Nexus3064 48x10GE
o2 Cyclades® ACS Advanced Console Server
pwr0 & pwr1 AP7941
?pwr0 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20
>pwr1 Servertech Horizontal Power Strip (8port) Part Number: CW-8H2-C20

Rack Physical and Wiring

hostname	switch	ipmi	oob	oobname	pwr	rack
o2.dfw.rg.net	sw0.35		1.8	o2			Cyclades® ACS-48
[equnix IX fiber]	0.46
r1.dfw.rg.net.2/2/0	0.07		2.15	r1		0.13-17	LAN
r1.dfw.rg.net.2/2/1	2.03		2.15	r2		0.13-17	NTT (VLAN 53)
r2.dfw.rg.net-0/0/1/0	2.05	0.08	2.3	r2		0.18.19	NTT VLAN 50
r2.dfw.rg.net.0/1/1	2.06		2.3			0.18.19	LAN
sw0.dfw.rg.net	2.47		2.13	sw0		0.28	Cicso 2960
sw2.dfw.rg.net	0.47		2.02	sw2	[^1 ^2]	0.30	Cisco 3064
raid0.dfw.rg.net	¹	0.32	2.44	raid0	0.5/1.5	0.36-38
vm0.dfw.rg.net	2.25 2.26(drbd)	2.20	2.5	vm0.dfw	pwr0b B1-11 pwr0r B1-9	0.40	SuperMicro
vm1.dfw.rg.net	2.27 2.28(drbd)	2.19	2.6	vm1.dfw	pwr0b B1-10 pwr0r B1-10	0.41	SuperMicro
vm2.dfw.rg.net	2.23 2.24(drbd)	2.21	2.6	vm2.dfw	pwr0b B1-09 pwr0r B1-11	0.42	Dell R630
atlas.dfw.rg.net	0.33				vm1.dfw USB

b0.noc.ietf.org	3.32	3.31	2.45	b0	0.4/1.4	1.26
sw3.dfw.rg.net	2.48		2.30	sw3	0.4/1.4	1.30	Cisco 3064
pwr1.red.dfw.rg.net	0.41		2.9	pwr1-red	subfl	26
pwr1.blue.dfw.rg.net	0.41		2.10	pwr1-blue	subfl	26
pwr0-blue.dfw.rg.net	0.42		2.16	pwr0-blue	subfl	-
pwr0-red.dfw.rg.net	0.42		2.17	pwr0-red	subfl	-

Removed

hostname	switch	ipmi	oob	oobname	pwr	rack
dallas01.mux.peering.usc.edu	0.38		2.46	usc	0.7/1.7	1.40
rtcolo.daedalus.com		xx	2.45	b0	xx	1.41

Cabling

vm	ipmi	e0	e1
vm0	sw2-20	sw2-25	sw2-26
vm1	sw2-19	sw2-27	sw2-28
vm2	sw2-21	sw2-23	sw2-24

ipmi	e0	e1	10g 0	10g 1
sw0-32	sw2-16	sw2-17	sw2-29	sw2-30

RGnet Cluster / 198.180.152.30 / 198.180.152.7 / 198.180.152.32

vm0.dfw.rg.net:/root# qm list  
      VMID NAME                 STATUS     MEM(MB)    DISK(GB) PID       
       111 cats.rg.net          running    2048          32.00 21557     
       112 cu0.dfw.rg.net       running    1024          12.00 368895    
       114 mail.rg.net          running    2048          32.00 368301    
       115 mon0.rg.net          running    4096         150.00 367610    
       117 wp.rg.net            running    2048          64.00 369294    
       118 ymbk.com             running    1024          16.00 65127     
       119 pfs.dfw.rg.net       running    1024           8.00 578419    
       122 ostrow.rg.net        running    2048          32.00 2216835   
vm1.dfw.rg.net:/home/randy> sudo qm list
      VMID NAME                 STATUS     MEM(MB)    DISK(GB) PID       
       102 git.rg.net           running    3072          32.00 118589    
       104 netsrv.dfw.rg.net    running    2046         100.00 119982    
       107 r3.dfw.rg.net        running    4096          32.00 121141    
       110 baleen.rg.net        running    2048          16.00 35309     
       113 krill.rg.net         running    2048          16.00 34511     
       116 routen8or.rg.net     running    2048         128.00 33328     
       120 a1000006.rg.net      running    512            8.00 3380241   
vm2.dfw.rg.net:/home/randy> sudo qm list
      VMID NAME                 STATUS     MEM(MB)    DISK(GB) PID       
       100 cache0.rg.net        running    3072          32.00 807710    
       101 ca.rg.net            running    5120         128.00 806001    
       103 icir.dfw.rg.net      running    8192         400.00 423985    
       105 nms.rg.net           running    2048          80.00 424708    
       106 petunia.hactrn.net   running    6144         120.00 302667    
       108 ran.psg.com          running    4096          32.00 321425    
       109 wireguard.rg.net     running    2048          10.00 325624    
       121 m0.rg.net            running    2560          32.00 3972

Hosts on vm0/1.dfx.rg.net

Instance	IP_Address	SysAdmin
a1000006.rg.net	198.180.152.21	Randy Bush randy@psg.com
baleen.rg.net	198.180.152.61	Randy Bush <randy@psg.com>
ca.rg.net	198.180.152.6	Randy Bush <randy@psg.com>
cache0.rg.net	198.180.152.33	Randy Bush <randy@psg.com>
cats.rg.net	198.180.152.58	Randy Bush <randy@psg.com>
cu0.dfw.rg.net	198.180.152.71	Steven M Bellovin <smb@cs.columbia.edu> John Koh <koh@…>
git.rg.net	198.180.152.8	Randy Bush <randy@psg.com>
icir.dfw.rg.net	198.180.152.57	Bill Marczak <billmar@gmail.com>
krill.rg.net	198.180.152.62	Randy Bush <randy@psg.com>
m0.rg.net	198.180.152.22	Randy Bush <randy@psg.com>
mail.rg.net	198.180.152.11	Randy Bush <randy@psg.com>
mon0.rg.net	198.180.152.62	Randy Bush <randy@psg.com>
netsrv.dfw.rg.net	198.180.152.15	Randy Bush <randy@psg.com>
nms.rg.net	198.180.152.9	Randy Bush <randy@psg.com>
ostrow.net	198.180.152.9	Ari Ostrow &ltaxo193@case.edu> Mark Allman &ltmallman@icir.org&gt
petunia.hactrn.net	198.180.152.200	Rob Austein <sra@hactrn.net>
pfs.rg.net	198.180.152.7	Randy Bush <randy@psg.com>
r3.dfw.rg.net	198.180.152.93	Randy Bush <randy@psg.com>
ran.psg.vom	198.180.152.18	Randy Bush <randy@psg.com>
routen8or.rg.net	198.180.152.63	Randy Bush <randy@psg.com>
wireguard.rg.net	198.180.152.16	Hans Kuhn <hans@flyingpoodle.com>
wp.rg.net	198.180.152.56	Randy Bush <randy@psg.com>
ymbk.com	198.180.152.11	Randy Bush <randy@psg.com>

vm0.dfx.rg.net vm1.dfx.rg.net

VMs are Supermicro A+ Server 1114S-WTRT Systems
With Supermicro H12SSW-NT Motherboards
With one AMD EPYCTM 7443P Single Socket Processor 24-Core 2.85GHz
With 256G of RAM each
And 2 Samsung 250GB 970 EVO+ Series - M.2 NVMe PCIe (<1DWPD) (MZ-V7S250B)
And 8 Samsung 960GB 883 DCT Series - 2.5" SSD SATA 6Gb/s (<1DWPD) (MZ-7LH960NE)
Serial Numbers: iX1210A-IXN S/N's A1-86827 and A1-86828

vm[0-1].dfx.rg.net Disk Drives

vm0.dfx.rg.net:/home/randy# for i in a b c d e f g h; do echo -n "sd$i "; smartctl -a /dev/sd$i | egrep "Serial|Model"; done
sda Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316039
sdb Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R107119
sdc Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316036
sdd Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316032
sde Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R303811
sdf Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316042
sdg Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R107124
sdh Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316040

vm1.dfx.rg.net:/home/randy#  for i in a b c d e f g h; do echo -n "sd$i "; smartctl -a /dev/sd$i | egrep "Serial|Model"; done
sda Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316035
sdb Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R303827
sdc Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316034
sdd Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R303829
sde Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R303777
sdf Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R303803
sdg Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R303764
sdh Device Model:     SAMSUNG MZ7LH960HAJR-00005
Serial Number:    S45NNA0R316031

vm2.dfw.rg.net

Dell 630
Service tag : 6JGVQD2
2 x E5-2670v3 2.30GHz 12-Core each
256G RAM
2 x 10G Ether SFP (eno1 eno2)
2 x 1G Ether Copper (eno3 eno4)
H730 RAID

raid0.dfw.rg.net

Dell R530
Service Tag: HQWPW52
PERC H730P RAID Controller, 2G B NV Cache, Mini card
2 x 4TB in md0 RAID1 as opsys
5 x 16TB in zfs
System Configuration Page

Drive Bays


Bay 0	Bay 2	Bay 4	Bay 6
Bay 1	Bay 3	Bay 5	Bay 7

Cable to Bay 5 may be bad 2023.07.16

raid0 Drives

smartctl -a -d megaraid,[0-7] /dev/bus/0
Look for smart scan errors with for i in 0 1 2 3 4 5 6 7; do echo -n "$i "; smartctl -a -d megaraid,$i /dev/bus/0 | grep 'Completed with'; echo; done


Bay	Manufacturer	Model	Serial ------------	Installed
0	Seagate	ST4000NM0025	ZC1BBMHX0000C943K87N	2024.07.28
1	Seagate	ST4000NM0005	Z4C056KT	2022.06.06
2	Western Digital	WD161KRYZ-01AGBB0	2PH9PMWT	2022.06.06
3	Western Digital	WD161KRYZ-01AGBB0	2PHA0RMT	2022.06.06
4	Western Digital	WD161KRYZ-01AGBB0	2PH9NJNT	2022.06.06
5	Western Digital	WD161KRYZ-01AGBBO	2CH669AP	2023.08.22
6	Seagate	ST16000NM000J-2TW103	ZR706BX0	2022.06.06

raid0.dfw.rg.net:/home/randy# for i in 0 1 2 3 4 5 6 7; do echo -n "$i  "; smartctl -a -d megaraid,$i /dev/bus/0 | grep 'Serial'; echo; done
0  Serial number:    Z4C056TC
1  Serial number:    Z4C056KT
2  Serial Number:    2PH9PMWT
3  Serial Number:    2PHA0RMT
4  Serial Number:    2PH9NJNT
5
6  Serial Number:    ZR706BX0
7  Serial Number:    2PH69HJJ

Disgusting Details

Power
    SO1-A:5C2.2-62/64
    SO2-A:5C2.2-66/68
    SO1-B:5D2.2-62/64
    SO2-B:5D2.2-66/68
    C-Side is Blue and that = "A" feed
    D-Side is Yellow and that = "B" feed
    [ SO2-B:5D2.2-66/68 will not be used ]

The demarc for the xconnects is as follows:
    pp01.dfw06
    137-138 M3 Future
    139-140 M4 Future
    141-142 M5 Equinix IX Cable ID #20466027 Terminated on CP:0104:1068601 ports 3/4
    143-144 M6 NTT/Verio r08.dllstx09.us.bb g0/6/0/29

Rge Equinix order info for the exchange fiber:
Ordering contact- Chris Morrow and/or CSM- Shazyra Irizarry
Technical contact – Randy Bush  <randy@psg.com>
IBX- DA1
Order:  1-97362053795
WOA-    3-97445043133 (IX Port Install)
WOA-    3-97448754941 (NCC Cable install)
Serial # 20466027 ; terminated on CP:0104:1068601 ports 3/4

---

Dallas Equinix Colo, Downstairs

Google demarc for the riser cable on the Equinix end is Suite 1034 Cage
00372 Cabinet 105

NTT/Verio TDI
Cage 220, Cab 101 FP#1, Port NA, SMF/SC.
r08.dllstx09.us.bb g0/6/0/29

Media: MMF
Connector: SC

---  ntt circuit ---

EQUINIX INC. - Cross Connect Order Complete - 817885.1.1.2
Completed On: 08-JUN-2009
IBX: DA1
Type of Order: Cross Connect
Cage: DA1::00080:EQUINIX.COM
Cabinet Number: 0101
A-Side Patch Panel/Module/Port: PP:0105:2288||1
Z-Side Customer and Cage Number: YOUTUBE, LLC     
     DA1::00372:YOUTUBE
Z-Side Cabinet Number: 0105
Carrier Circuit ID(if applicable): EQUINIX SANDBOX
Z-Side Patch Panel/Module/Port: VP:0105:69862||143
Cable ID: 1234912

EQUINIX INC. - Cross Connect Order Complete - 817885.2.1.2
Completed On: 05-JUN-2009
IBX: DA1
Type of Order: Cross Connect
Cage: DA1::00080:EQUINIX.COM
Cabinet Number: 0101
A-Side Patch Panel/Module/Port: PP:0105:74459||3
Z-Side Customer and Cage Number: NTT AMERICA, INC.
     DA1::00220:NTT AMERICA, INC. (Verio)
Z-Side Cabinet Number: 0101
Carrier Circuit ID(if applicable): EQUINIX SANDBOX
Z-Side Patch Panel/Module/Port: PP:0101:2375||15
Cable ID: 1234913

For both crossconnects, even though the "Cabinet Number" field shows
"0101", the actual Equinix A-Side cab is "0105" as listed in the panel
info.  And to reduce cabling and points of failure, the IBX site
technicians said that they would be connecting both crossconnects
directly into each other rather than with an intermediate patch cable.

--- equinix exchange ---

csw7.da1, gig 0/35, Sandbox/RGNet, IX, circuit id 1236454

Routers

[]

Tapping BGP Feeds

[]

Old Google PoP

We were in Google POP in Via West

Viawest/Dataside
1950 Stemmons Freeway, Suite 2033
Dallas TX 75027
+1-214-231-0162
Mark for "inside delivery"

Google PoP DFW06
Rack Location: 6.3
Rack: ESW E-00589-01, Black Caster and Mounting Rails, No Door, 7Ft.

Devices Removed

hostname	switch	oob	oobname	pwr	rack
r0.dfw.rg.net-rp0	0.45	0.01	r00	subfl	03-13
r0.dfw.rg.net-rp0mgt	0.33				03-13
r0.dfw.rg.net-rp1	0.??	0.02	r01		03-13
o1.dfw.rg.net	0.40	0.aux		1.1	0.22-23
r0.dfw.rg.net-rp1mgt	0.34				03-13
[r0.2/0/4 LAN]	0.04				03-13
[r0.2/0/5 ntt fiber]	0.05				03-13
[r0.2/0/6 spare]	0.06				03-13
[r0.2/0/7 equinix ix]	0.15				03-13
sw1.dfw.rg.net		0.13	sw1	²	0.35

DEAD raid0.dfw.rg.net

Mobo Supermicro X7SB4/E IPMI AOC-IPMI20-E

raid0 disk drive physical order markings on drive carriers


00	10	20	30
01	11	21	31
02	12	22	32
03	13	23	33

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do
   echo -n "1/$i  ";
   smartctl -i -d hpt,1/$i /dev/hptrr | grep ^Serial;
   done

1/1 Serial Number: WD-WMAZA9378538 1/2 Serial Number: V1KVXSXG 1/3 Serial Number: YGGU1YTA 1/4 Serial Number: WD-WMAZA9520754 1/5 Serial Number: WD-WCAVY0114154 1/6 Serial Number: WD-WMC5C0D9L6HV 1/7 Serial Number: WD-WCC6N5UY8H5P 1/8 Serial Number: WD-WCC300759632 1/9 Serial Number: WD-WCC6N6DDJ7DT 1/10 Serial Number: WD-WCC6N6VTE2NF 1/11 Serial Number: WD-WMC5C0D8Z7FV 1/12 Serial Number: WD-WMC6N0J6S9AV 1/13 Serial Number: WD-WCC4MLULA4S4 1/14 Serial Number: WD-WMC6N0F8FU0U 1/15 Serial Number: WD-WCAVY0090692 1/16 Serial Number: WD-WCC6N5CLHYSS

Bay	ID	Drive	Name	Serial Number
20	1/1	da0	JBOD-1-1	WD-WMAZA9378538
21	1/2	da1	JBOD-1-2	V1KVXSXG
22	1/3	da2	JBOD-1-3	WD-WCC300755734
23	1/4	da3	JBOD-1-4	WD-WMAZA9520754
00	1/5	da4	JBOD-1-5	WD-WCAVY0114154
01	1/6	da5	JBOD-1-6	WD-WMC5C0D9L6HV
02	1/7	da6	JBOD-1-7	WD-WCC6N5UY8H5P
03	1/8	da7	JBOD-1-8	WD-WCC300759632
30	1/9	da8	JBOD-1-9	WD-WCC6N6DDJ7DT
31	1/10	da9	JBOD-1/10	WD-WCC6N6VTE2NF
32	1/11	da10	JBOD-1/11	WD-WMC5C0D8Z7FV
33	1/12	da11	JBOD-1/12	WD-WMC6N0J6S9AV
10	1/13	da12	JBOD-1/13	WD-WCC4MLULA4S4
11	1/14	da13	JBOD-1/14	WD-WMC6N0F8FU0U
12	1/15	da14	JBOD-1/15	WD-WCAVY0090692
13	1/16	da15	JBOD-1/16	WD-WCC6N5CLHYSS

da1 MK0171YFKMA4AG replaced 2022.05.08
da12 MK0171YFKMA59G replaced 2022.04.12
da8 WD-WCAVY0111166 replaced 2021.11.22
da10 WD-WCAVY0127426 replaced 2016.09.15
da1 S2F06GX5 replaced 2016.01.2?
da2 WD-WCAVY0112831 replaced 2014.05.12
da5 WD-WCAVY0107135 replaced 2017.03.25
da13 W1E0B4NJ replaced 2017.06.xx
da6 WD-WCAVY0127865 replaced 2018.01.07
da6 WD-WMC6N0H8K48M replaced 2021.01.16
da9 WD-WCAVY0117958 replaced 2021.03.15
da15 WD-WCC4M5VJ4UK3 replaced 2021.06.18
da9 WD-WMAY04440674 replaced 2021.08.27

Decommissioned

dfw0 1ru compute server SuperMicro SYS-6015W-NTRB w/ X7DWU
dfw1 3ru big raid server SuperMicro CSE-836TQ-R800B w/ X7SBE
dfw2 2ru Junosphere virtualization server

DEAD dfw0.psg.com

dfw0 drives for i in 4 5 6 7; do echo /dev/ad$i; smartctl -a /dev/ad$i | egrep 'Model|Serial'; done


Bay	Model	Serial

ada0	ST31000340NS	9QJ3Q29E	1TB
ada1	WD20EADS-00R6B0	WCAVY0388932	2TB
ada2	WD2005FBYZ-01YCBB1	WMC6N0F2KZU6	2TB
ada3	ST31000340NS	9QJ3M5N4	1TB

dfw0 disk drive physical order

ada4	ada6	ada5	ada7
ada0	ada2	ada1	ada3

logical drive labels

label/disk00	ada0s3
label/disk01	ada1s3
label/disk02	ada2s1
label/disk03	ada3s1

DEAD nfs0.dfw.rg.net

MoBo

DEAD nfs0.dfw.rg.net Disk Drives

for i in 0 1 2 3 4 5 6 7; do tw_cli /c2/p$i show serial; done

Port	Model	Serial	Size	Date
p0	ST4000DM000-1F2168	Z300L98C	4TB	don't know
p1	WD4000F9YZ-09N20L1	WD-WCC5D1DS4012	4TB	2019.04.07
p2	WD60EFRX-68L0BN1	WD-WX41DA57H7UJ	6TB	2018.03.09
p3	WD4000F9YZ-09N20L1	WD-WCC5D2KHHYE8	4TB	2016.06.09
p4	ST4000DM000-1F2168	Z300KBMM	4TB	don't know
p5	ST4000DM000-1F2168	Z300KBQN	4TB	don't know
p6	ST4000DM000-1F2168	Z300L8J5	4TB	don't know
p7	WD4000F9YZ-09N20L1	WD-WMC5D0D7V0L9	4TB	2015.12.13

p1	ST4000DM000-1F2168	Z300FKES	4TB	removed 2019.04.07
p2	ST4000DM000-1F2168	Z300L9T3	4TB	removed 2018.03.09
incoming	HUS724040ALS640	PBK4K70X	4TB	2015.12.18

Physical Drive layout


4	5	6	7
0	1	2	3

2025.02.01

raid0.dfw ethers are connected to sw2.dfw as follows: ↩︎
RGnet VM servers' ethers are connected to sw2.dfw as follows: ↩︎

24 KiB Raw Blame History Unescape Escape