HC1: Operations SIG 9 Sep 2024

Agenda

What: Meeting to Discuss Improving Node Operations as part of the HC1: Operations SIG

When: 9 Sep 2024 @ 6PM EST

Where: https://matrix.to/#/#sig-operations:hc1.chat

Chair: 0x3639

Agenda:

1. Discuss follow Up items from previous meeting
2. Document action items
3. Establish next meeting

If you want to attend please respond (or DM) with your full matrix username and I will invite you to the group. No FUD, anger or BS allowed.

Pre-meeting Notes

0x3639

• moved ARM support to next phase
• added grafana branch and added --grafana flag to install grafana with node exporter setup
• added /dashboard folder and added the znnd node dashboard George created
• fixed ascii art formatting
• added DRAFT README.md
• added noninteractive support for --deploy
• noninteractive support for --grafana not working
• merged the grafana branch into main

George

• created grafana dashboard that uses infinity plugin to query stats rpc endpiont and show sync state

Coinselor

• updated README
• should we add --backup to save a bootstrap file ?
• should we add --uninstall to undo script changes ?

Minutes

Mon, Sep 9, 2024, 17:00:14 - deeznnutz: === Starting Meeting | 9 Sept 2024 @ 6PM EST===

Mon, Sep 9, 2024, 17:00:18 - georgezgeorgez: Hello hello

Mon, Sep 9, 2024, 17:00:21 - deeznnutz: hello!!

Mon, Sep 9, 2024, 17:00:44 - deeznnutz: I'm trying the meeting start to make it easier to make the meeting minutes

Mon, Sep 9, 2024, 17:01:53 - georgezgeorgez: It really needs a way to export by timeframe

Mon, Sep 9, 2024, 17:02:04 - georgezgeorgez: But anyways

Mon, Sep 9, 2024, 17:02:27 - deeznnutz: agree. I send a dm to coinselor. Should we start?

Mon, Sep 9, 2024, 17:02:36 - georgezgeorgez: Yes let's start

Mon, Sep 9, 2024, 17:02:46 - georgezgeorgez: So I think we had a pretty productive last 2 weeks

Mon, Sep 9, 2024, 17:02:54 - georgezgeorgez: Or at least, we have steady progress

Mon, Sep 9, 2024, 17:03:03 - deeznnutz: ya, agree. I've got a little summary

Mon, Sep 9, 2024, 17:03:06 - deeznnutz: Since our last meeting we've made good progress. I'll summarize the recent updates:

• Moved ARM support to a future release
• Added custom znnd Dashboard for integration with the Infinity Plugin. It's in the /Dashboard folder
• Added --grafana flag to install grafana, prometheus, node exporter and the standard node exporter dashboard
• Added noninteractive support for the --deploy flag
• Added DRAFT README.md
• Noninteractive support for --grafana is not working.

Did I miss anything?

Mon, Sep 9, 2024, 17:03:24 - georgezgeorgez: coinselor

Mon, Sep 9, 2024, 17:03:34 - georgezgeorgez: his chatgpt made docs?

Mon, Sep 9, 2024, 17:03:42 - georgezgeorgez: if it's actually useful

Mon, Sep 9, 2024, 17:03:54 - georgezgeorgez: not bad as a starting point at the very least

Mon, Sep 9, 2024, 17:03:59 - deeznnutz: lol. ya, I asked it to make me a readme.md and it was pretty good. So I added it as a draft

Mon, Sep 9, 2024, 17:04:52 - coinselor: I'm here ^^

Mon, Sep 9, 2024, 17:05:02 - georgezgeorgez: I want to zoom out and go back to the three high level things:

1. Make it easy to deploy
2. Help people monitor their system
3. Help people support (diagnostics)

Mon, Sep 9, 2024, 17:05:11 - georgezgeorgez: We're hitting a baseline maturity for 1 and 2

Mon, Sep 9, 2024, 17:05:33 - georgezgeorgez: I want to start inviting people to beta test out the scripts/stack

Mon, Sep 9, 2024, 17:05:51 - georgezgeorgez: And then next meeting, if we can have a polished version, that would be great

Mon, Sep 9, 2024, 17:06:00 - georgezgeorgez: We should start thinking about 3

Mon, Sep 9, 2024, 17:06:07 - deeznnutz: quickly.

Mon, Sep 9, 2024, 17:06:09 - deeznnutz: Potential next steps before we request community testing:

• Setup go.zenon.sh for installation - Something like curl -s https://go.zenon.sh/go-zenon.sh | bash
• Improve / finish the "docs" and/or README.md
• Fix the noninteractive support for --grafana

Mon, Sep 9, 2024, 17:06:19 - deeznnutz: What do you think about these before asking for testing?

Mon, Sep 9, 2024, 17:07:16 - georgezgeorgez: I think the first bullet point isn't too critical for testing. Curling a GitHub raw asset is fine for now. For general release, yeah, a good URL would be nice.

Mon, Sep 9, 2024, 17:07:31 - georgezgeorgez: For 2, we need enough docs, but doesn't have to be perfect

Mon, Sep 9, 2024, 17:07:39 - coinselor: Help people support (diagnostics) -> This is kinda vague. Can you be more specific?

Mon, Sep 9, 2024, 17:08:12 - georgezgeorgez: Sure, so right now HC1, and in particular deeznnutz, does a lot of 1-1 sessions

Mon, Sep 9, 2024, 17:08:26 - georgezgeorgez: With community members to help get their infra up and keep it running

Mon, Sep 9, 2024, 17:08:54 - georgezgeorgez: And there's usually some back and forth in terms of asking questions

Mon, Sep 9, 2024, 17:09:06 - georgezgeorgez: A lot of that can be streamlined

Mon, Sep 9, 2024, 17:09:21 - georgezgeorgez: If for example, there was a tool that could fetch info about the node and zip it up

Mon, Sep 9, 2024, 17:09:49 - georgezgeorgez: So they can send it over to whoever is helping or in a bug report

Mon, Sep 9, 2024, 17:09:54 - georgezgeorgez: Especially again, once we get into testnets

Mon, Sep 9, 2024, 17:09:59 - georgezgeorgez: 0x linked a tool earlier

Mon, Sep 9, 2024, 17:09:59 - georgezgeorgez: sos

Mon, Sep 9, 2024, 17:10:36 - deeznnutz: Each session has the same questions over and over and it takes like 15 minutes to get the info. Automating that would be so helpful

Mon, Sep 9, 2024, 17:11:04 - deeznnutz: Simple things like, is go-zenon running

Mon, Sep 9, 2024, 17:11:06 - deeznnutz: lol

Mon, Sep 9, 2024, 17:11:18 - deeznnutz: Do you have enough hard drive space

Mon, Sep 9, 2024, 17:11:24 - deeznnutz: Is the swap file set

Mon, Sep 9, 2024, 17:12:06 - deeznnutz: Back to the public testing, getting the noninteractive install working is actually important, b/c everyone asks what to do at those screens

Mon, Sep 9, 2024, 17:12:18 - georgezgeorgez: I think we should do some more research into the tool and see if there are any alternatives

Mon, Sep 9, 2024, 17:13:09 - georgezgeorgez: 0x, do you have the link to that sos tool?

Mon, Sep 9, 2024, 17:13:26 - georgezgeorgez: I think it had a plugin architecture, and we could possibly write our own plugin

Mon, Sep 9, 2024, 17:13:27 - deeznnutz: I have time this week to finish up the noninteractive fixes and finish the readme.md. From there I think we can ask for testers. Agree?

Mon, Sep 9, 2024, 17:13:36 - coinselor: https://github.com/sosreport/sos

Mon, Sep 9, 2024, 17:13:38 - coinselor: This?

Mon, Sep 9, 2024, 17:13:55 - deeznnutz: ya, that one

Mon, Sep 9, 2024, 17:14:15 - deeznnutz: It has a nice plugin system too. We can write one for znnd.

Mon, Sep 9, 2024, 17:14:25 - coinselor: "everyone asks what to do at those screens" - do you mean the znncontroller interactive screen?

Mon, Sep 9, 2024, 17:14:33 - deeznnutz: I would like to get the config.json without the producer password.

Mon, Sep 9, 2024, 17:14:38 - georgezgeorgez: I think non-critical for the general release but we've talked about it here is historical timeseries for sync status

Mon, Sep 9, 2024, 17:14:50 - georgezgeorgez: I think that will be our first entry into performance testing

Mon, Sep 9, 2024, 17:15:00 - deeznnutz: <@coinselor:zenon.chat ""everyone asks what to do at tho..."> no, when you install an app in Linux it asks if you want to restart a service.

Mon, Sep 9, 2024, 17:15:12 - georgezgeorgez: It would be good to compare the sync charts of nodes with and without vilkris's updates

Mon, Sep 9, 2024, 17:15:13 - deeznnutz: That purple screen.

Mon, Sep 9, 2024, 17:15:45 - coinselor: lol ya, that purple screen, I always press enter I don't even know what I'm looking at. Looks like some BIOS from 1980

Mon, Sep 9, 2024, 17:16:14 - deeznnutz: Exactly. I want to auto-select enter for those.

Mon, Sep 9, 2024, 17:16:20 - coinselor: But can that be skipped? Seems like some low-level VM stuff

Mon, Sep 9, 2024, 17:16:33 - deeznnutz: Yes, it can be auto-selected

Mon, Sep 9, 2024, 17:16:48 - deeznnutz: You tell it to accept all defaults and make the installation noninteractive.

Mon, Sep 9, 2024, 17:17:48 - georgezgeorgez: Noninteractive will be more important for multinode setups

Mon, Sep 9, 2024, 17:18:03 - georgezgeorgez: I'm guessing most people will still be SSH'ing to their box to install?

Mon, Sep 9, 2024, 17:18:16 - deeznnutz: ya, I think so

Mon, Sep 9, 2024, 17:18:36 - deeznnutz: I think we can target testing this week

Mon, Sep 9, 2024, 17:19:06 - deeznnutz: Improve readme.md and noninteractive, and we can test I think

Mon, Sep 9, 2024, 17:19:09 - georgezgeorgez: Do we want to set a closer deadline to get some of these basics done? Then send out a call for beta testers?

Mon, Sep 9, 2024, 17:19:30 - deeznnutz: ya, I'd like to start testing by Friday this week

Mon, Sep 9, 2024, 17:21:04 - coinselor: My readme is chatgpt'd but I edited/proofread. I don't know what else could be added other than more content as we add more functionality

Mon, Sep 9, 2024, 17:21:19 - coinselor: It's on my forked repo tho

Mon, Sep 9, 2024, 17:21:31 - deeznnutz: Maybe add a tl;dr getting started?

Mon, Sep 9, 2024, 17:21:36 - georgezgeorgez: Make an MR

Mon, Sep 9, 2024, 17:22:22 - coinselor: OK, I will. Maybe 0x can edit and add stuff from his chatgpt readme XD

Mon, Sep 9, 2024, 17:24:20 - deeznnutz: Regarding next steps and roadmap, I typed up a few things. Maybe we should prioritize?

Mon, Sep 9, 2024, 17:24:23 - deeznnutz: Roadmap

• ARM support
• Automate the installation and setup of the znnd Dashboard
• Troubleshooting script to send logs to devs
• Troubleshoot ports and accessibility (are my ports open)
• Deploy testnet
• Add --uninstall flag
• Add --backup flag (I have a version of this script already)

Mon, Sep 9, 2024, 17:25:27 - deeznnutz: What is a higher priority? Automate the installation of the znnd dash or troubleshooting "stuff"?

Mon, Sep 9, 2024, 17:25:57 - georgezgeorgez: I think we should prioritize a single happy path that is useful to most people first

Mon, Sep 9, 2024, 17:26:03 - coinselor: I can get a few aliens to commit to help us test it

Mon, Sep 9, 2024, 17:26:25 - georgezgeorgez: What would enable people to actually start using the script and running their nodes with it?

Mon, Sep 9, 2024, 17:26:43 - georgezgeorgez: The priority should be first to get some real users, and then after that, they can help us prioritize

Mon, Sep 9, 2024, 17:26:45 - deeznnutz: I like the znnd dash. It's actually pretty useful

Mon, Sep 9, 2024, 17:27:14 - deeznnutz: So IMO the cool dash will be more useful to users than troubleshooting

Mon, Sep 9, 2024, 17:27:48 - georgezgeorgez: Yeah, I think we can get 1 and 2 to a good spot before focusing on 3

Mon, Sep 9, 2024, 17:27:52 - sugoi joined the room

Mon, Sep 9, 2024, 17:27:56 - georgezgeorgez: Usable spot

Mon, Sep 9, 2024, 17:28:07 - sugoi: Hi guys

Mon, Sep 9, 2024, 17:28:15 - georgezgeorgez: Hi, welcome to the operations SIG

Mon, Sep 9, 2024, 17:29:25 - georgezgeorgez: Not sure how much you have been following, but we've been working on a znnd installation script and a dashboard to help people understand their node

Mon, Sep 9, 2024, 17:29:51 - deeznnutz: Does anyone have time to work on the auto-deploy of the znnd dash? You can use some code already written in the script to deploy the dash w/ the API

Mon, Sep 9, 2024, 17:29:57 - coinselor: <@georgezgeorgez:hc1.chat "What would enable people to actu..."> I would personally start using it because it just automates a lot of the stuff. Grafana dashboard and other stuff will also come in handy if someone asks for system information.

Mon, Sep 9, 2024, 17:30:18 - sugoi: <@georgezgeorgezhc1.chat "not sure how much you have been ..."> Trying to read up now. Thanks for the tl;dr!

Mon, Sep 9, 2024, 17:30:39 - deeznnutz: So the dash requires:

• Install the infinity plugin
• Set the default datasource (127.0.0.1)
• Import the dashboard w/ the API

Mon, Sep 9, 2024, 17:30:49 - georgezgeorgez: Yes, we will likely be our own first users haha

Mon, Sep 9, 2024, 17:31:40 - georgezgeorgez: <@deeznnutz:zenon.chat "so the dash requires:"> Let's target this for Friday?

Mon, Sep 9, 2024, 17:31:41 - deeznnutz: It took me a while to figure out the import dash API. It had a bunch of size limits

Mon, Sep 9, 2024, 17:32:19 - deeznnutz: <@georgezgeorgez:hc1.chat "let's target this for friday?"> I can get the readme and noninteractive done by this Friday. Not sure about the znnd dash

Mon, Sep 9, 2024, 17:32:28 - georgezgeorgez: I can work on it

Mon, Sep 9, 2024, 17:32:46 - deeznnutz: OK, if you have time, then I'm sure this Friday is possible.

Mon, Sep 9, 2024, 17:33:35 - deeznnutz: Just reuse this: https://github.com/go-zenon/go/blob/82ba64fbee715ad468c878a06cd16ac6258ccad1/grafana.sh#L119

Mon, Sep 9, 2024, 17:34:01 - georgezgeorgez: Yup.

0x, we can do it after this meeting and throughout the week, but let's get more GitHub issues created for all this

Mon, Sep 9, 2024, 17:34:11 - deeznnutz: That node exporter dash was very large and I kept getting file size errors

Mon, Sep 9, 2024, 17:34:15 - georgezgeorgez: I see you've created a couple already

Mon, Sep 9, 2024, 17:34:24 - coinselor: Yeah +1 to issues, break it down into tasks we can pick up

Mon, Sep 9, 2024, 17:34:38 - coinselor: For instance, I have no idea what this entails: Troubleshooting script to send logs to devs

Mon, Sep 9, 2024, 17:34:40 - deeznnutz: Yep. Makes sense

Mon, Sep 9, 2024, 17:35:25 - georgezgeorgez: Cool, so we have short-term goals for this week and also some other things on hold to revisit once we get it out for beta testing

Mon, Sep 9, 2024, 17:35:32 - georgezgeorgez: Anything else we want to go over?

Mon, Sep 9, 2024, 17:35:37 - georgezgeorgez: Or just ideas?

Mon, Sep 9, 2024, 17:35:58 - deeznnutz: Awesome. I can also work on adding/cleaning up the issues this week.

Mon, Sep 9, 2024, 17:36:01 - deeznnutz: That's all I had!

Mon, Sep 9, 2024, 17:36:26 - georgezgeorgez: Let's get another meeting on the cal

Mon, Sep 9, 2024, 17:36:32 - deeznnutz: I've gotten feedback from other devs watching this group and think we are making good progress on working together

Mon, Sep 9, 2024, 17:36:50 - deeznnutz: So I'm happy we are making these small steps together

Mon, Sep 9, 2024, 17:37:01 - coinselor: Just wishful thinking but would be badass to add --public, --sentry, --pillar

Mon, Sep 9, 2024, 17:37:01 - georgezgeorgez: I think even though it's all over chat, having regular times for the meetings helps moves things along

Mon, Sep 9, 2024, 17:37:09 - deeznnutz: 23 Sept @ 6PM EST?

Mon, Sep 9, 2024, 17:37:21 - georgezgeorgez: Message deleted

Mon, Sep 9, 2024, 17:37:33 - georgezgeorgez: <@coinselor:zenon.chat "just wishful thinking but would ..."> Likely on our path

Mon, Sep 9, 2024, 17:38:29 - georgezgeorgez: <@deeznnutz:zenon.chat "I've gotten feedback from other ..."> My goal with this SIG so far has been to bootstrap some sort of steady collaborative effort

Mon, Sep 9, 2024, 17:39:05 - coinselor: Particularly for the upcoming sporks, I know there's a way to like upgrade the pillar with no downtime. I'm not doing that unless this script can handle it 😂 no worth the effort, I rather miss the momentums lol

Mon, Sep 9, 2024, 17:39:21 - georgezgeorgez: Once we get some credibility, need to start figuring out incentives. Then the pace can pick up and really be driven by the community/userbase

Mon, Sep 9, 2024, 17:39:32 - georgezgeorgez: <@coinselor:zenon.chat "particularly for the upcoming sp..."> Bro, make an issue

Mon, Sep 9, 2024, 17:39:47 - georgezgeorgez: We need to track this stuff, even if it will be a bit before we get to it

Mon, Sep 9, 2024, 17:40:02 - georgezgeorgez: It helps too, because other people might chime in and say it's important for them too

Mon, Sep 9, 2024, 17:40:03 - deeznnutz: ya, we have a feature request tag in there

Mon, Sep 9, 2024, 17:40:12 - georgezgeorgez: And that will help us determine our priority

Mon, Sep 9, 2024, 17:40:35 - deeznnutz: Awesome. Is everyone OK with 23 Sept @ 6PM EST?

Mon, Sep 9, 2024, 17:40:39 - georgezgeorgez: But again, I'm pushing the saying:

Mon, Sep 9, 2024, 17:40:41 - georgezgeorgez: Incentivization is Prioritization

Mon, Sep 9, 2024, 17:41:05 - georgezgeorgez: My idea is that SIGs define the frontier of development. What is the next possible thing we can do?

Mon, Sep 9, 2024, 17:41:15 - georgezgeorgez: And the community helps prioritize it by setting incentives

Mon, Sep 9, 2024, 17:41:29 - deeznnutz: Makes perfect sense!

Mon, Sep 9, 2024, 17:41:51 - coinselor: OK, next step is to populate GitHub then

Mon, Sep 9, 2024, 17:41:53 - coinselor: Take it from there

Mon, Sep 9, 2024, 17:42:13 - deeznnutz: Cool. Thanks everyone. I'm around all week.

Mon, Sep 9, 2024, 17:42:22 - georgezgeorgez: Thank you everyone, good meeting

Mon, Sep 9, 2024, 17:42:29 - coinselor: gn!!

Mon, Sep 9, 2024, 17:42:33 - deeznnutz: gn