HC1: Operations SIG 9 Sep 2024: Difference between revisions
→Minutes: Added meeting minutes |
m George moved page Operations SIG 9 Sep 2024 to HC1: Operations SIG 9 Sep 2024 |
||
(No difference)
|
Latest revision as of 02:05, 20 September 2024
Agenda
What: Meeting to Discuss Improving Node Operations as part of the HC1: Operations SIG
When: 9 Sep 2024 @ 6PM EST
Where: https://matrix.to/#/#sig-operations:hc1.chat
Chair: 0x3639
Agenda:
- Discuss follow Up items from previous meeting
- Document action items
- Establish next meeting
If you want to attend please respond (or DM) with your full matrix username and I will invite you to the group. No FUD, anger or BS allowed.
Pre-meeting Notes
- moved ARM support to next phase
- added grafana branch and added
--grafana
flag to install grafana with node exporter setup - added
/dashboard
folder and added the znnd node dashboard George created - fixed ascii art formatting
- added DRAFT
README.md
- added noninteractive support for
--deploy
- noninteractive support for
--grafana
not working - merged the grafana branch into main
- created grafana dashboard that uses infinity plugin to query stats rpc endpiont and show sync state
- updated README
- should we add
--backup
to save a bootstrap file ? - should we add
--uninstall
to undo script changes ?
Minutes
Mon, Sep 9, 2024, 17:00:14 - deeznnutz: === Starting Meeting | 9 Sept 2024 @ 6PM EST===
Mon, Sep 9, 2024, 17:00:18 - georgezgeorgez: Hello hello
Mon, Sep 9, 2024, 17:00:21 - deeznnutz: hello!!
Mon, Sep 9, 2024, 17:00:44 - deeznnutz: I'm trying the meeting start to make it easier to make the meeting minutes
Mon, Sep 9, 2024, 17:01:53 - georgezgeorgez: It really needs a way to export by timeframe
Mon, Sep 9, 2024, 17:02:04 - georgezgeorgez: But anyways
Mon, Sep 9, 2024, 17:02:27 - deeznnutz: agree. I send a dm to coinselor. Should we start?
Mon, Sep 9, 2024, 17:02:36 - georgezgeorgez: Yes let's start
Mon, Sep 9, 2024, 17:02:46 - georgezgeorgez: So I think we had a pretty productive last 2 weeks
Mon, Sep 9, 2024, 17:02:54 - georgezgeorgez: Or at least, we have steady progress
Mon, Sep 9, 2024, 17:03:03 - deeznnutz: ya, agree. I've got a little summary
Mon, Sep 9, 2024, 17:03:06 - deeznnutz: Since our last meeting we've made good progress. I'll summarize the recent updates:
- Moved ARM support to a future release
- Added custom
znnd
Dashboard for integration with the Infinity Plugin. It's in the/Dashboard
folder - Added
--grafana
flag to install grafana, prometheus, node exporter and the standard node exporter dashboard - Added noninteractive support for the
--deploy
flag - Added DRAFT
README.md
- Noninteractive support for
--grafana
is not working.
Did I miss anything?
Mon, Sep 9, 2024, 17:03:24 - georgezgeorgez: coinselor
Mon, Sep 9, 2024, 17:03:34 - georgezgeorgez: his chatgpt made docs?
Mon, Sep 9, 2024, 17:03:42 - georgezgeorgez: if it's actually useful
Mon, Sep 9, 2024, 17:03:54 - georgezgeorgez: not bad as a starting point at the very least
Mon, Sep 9, 2024, 17:03:59 - deeznnutz: lol. ya, I asked it to make me a readme.md and it was pretty good. So I added it as a draft
Mon, Sep 9, 2024, 17:04:52 - coinselor: I'm here ^^
Mon, Sep 9, 2024, 17:05:02 - georgezgeorgez: I want to zoom out and go back to the three high level things:
- Make it easy to deploy
- Help people monitor their system
- Help people support (diagnostics)
Mon, Sep 9, 2024, 17:05:11 - georgezgeorgez: We're hitting a baseline maturity for 1 and 2
Mon, Sep 9, 2024, 17:05:33 - georgezgeorgez: I want to start inviting people to beta test out the scripts/stack
Mon, Sep 9, 2024, 17:05:51 - georgezgeorgez: And then next meeting, if we can have a polished version, that would be great
Mon, Sep 9, 2024, 17:06:00 - georgezgeorgez: We should start thinking about 3
Mon, Sep 9, 2024, 17:06:07 - deeznnutz: quickly.
Mon, Sep 9, 2024, 17:06:09 - deeznnutz: Potential next steps before we request community testing:
- Setup go.zenon.sh for installation - Something like
curl -s https://go.zenon.sh/go-zenon.sh | bash
- Improve / finish the "docs" and/or
README.md
- Fix the noninteractive support for
--grafana
Mon, Sep 9, 2024, 17:06:19 - deeznnutz: What do you think about these before asking for testing?
Mon, Sep 9, 2024, 17:07:16 - georgezgeorgez: I think the first bullet point isn't too critical for testing. Curling a GitHub raw asset is fine for now. For general release, yeah, a good URL would be nice.
Mon, Sep 9, 2024, 17:07:31 - georgezgeorgez: For 2, we need enough docs, but doesn't have to be perfect
Mon, Sep 9, 2024, 17:07:39 - coinselor: Help people support (diagnostics) -> This is kinda vague. Can you be more specific?
Mon, Sep 9, 2024, 17:08:12 - georgezgeorgez: Sure, so right now HC1, and in particular deeznnutz, does a lot of 1-1 sessions
Mon, Sep 9, 2024, 17:08:26 - georgezgeorgez: With community members to help get their infra up and keep it running
Mon, Sep 9, 2024, 17:08:54 - georgezgeorgez: And there's usually some back and forth in terms of asking questions
Mon, Sep 9, 2024, 17:09:06 - georgezgeorgez: A lot of that can be streamlined
Mon, Sep 9, 2024, 17:09:21 - georgezgeorgez: If for example, there was a tool that could fetch info about the node and zip it up
Mon, Sep 9, 2024, 17:09:49 - georgezgeorgez: So they can send it over to whoever is helping or in a bug report
Mon, Sep 9, 2024, 17:09:54 - georgezgeorgez: Especially again, once we get into testnets
Mon, Sep 9, 2024, 17:09:59 - georgezgeorgez: 0x linked a tool earlier
Mon, Sep 9, 2024, 17:09:59 - georgezgeorgez: sos
Mon, Sep 9, 2024, 17:10:36 - deeznnutz: Each session has the same questions over and over and it takes like 15 minutes to get the info. Automating that would be so helpful
Mon, Sep 9, 2024, 17:11:04 - deeznnutz: Simple things like, is go-zenon
running
Mon, Sep 9, 2024, 17:11:06 - deeznnutz: lol
Mon, Sep 9, 2024, 17:11:18 - deeznnutz: Do you have enough hard drive space
Mon, Sep 9, 2024, 17:11:24 - deeznnutz: Is the swap file set
Mon, Sep 9, 2024, 17:12:06 - deeznnutz: Back to the public testing, getting the noninteractive install working is actually important, b/c everyone asks what to do at those screens
Mon, Sep 9, 2024, 17:12:18 - georgezgeorgez: I think we should do some more research into the tool and see if there are any alternatives
Mon, Sep 9, 2024, 17:13:09 - georgezgeorgez: 0x, do you have the link to that sos tool?
Mon, Sep 9, 2024, 17:13:26 - georgezgeorgez: I think it had a plugin architecture, and we could possibly write our own plugin
Mon, Sep 9, 2024, 17:13:27 - deeznnutz: I have time this week to finish up the noninteractive fixes and finish the readme.md
. From there I think we can ask for testers. Agree?
Mon, Sep 9, 2024, 17:13:36 - coinselor: https://github.com/sosreport/sos
Mon, Sep 9, 2024, 17:13:38 - coinselor: This?
Mon, Sep 9, 2024, 17:13:55 - deeznnutz: ya, that one
Mon, Sep 9, 2024, 17:14:15 - deeznnutz: It has a nice plugin system too. We can write one for znnd.
Mon, Sep 9, 2024, 17:14:25 - coinselor: "everyone asks what to do at those screens" - do you mean the znncontroller interactive screen?
Mon, Sep 9, 2024, 17:14:33 - deeznnutz: I would like to get the config.json
without the producer password.
Mon, Sep 9, 2024, 17:14:38 - georgezgeorgez: I think non-critical for the general release but we've talked about it here is historical timeseries for sync status
Mon, Sep 9, 2024, 17:14:50 - georgezgeorgez: I think that will be our first entry into performance testing
Mon, Sep 9, 2024, 17:15:00 - deeznnutz: <@coinselor:zenon.chat ""everyone asks what to do at tho..."> no, when you install an app in Linux it asks if you want to restart a service.
Mon, Sep 9, 2024, 17:15:12 - georgezgeorgez: It would be good to compare the sync charts of nodes with and without vilkris's updates
Mon, Sep 9, 2024, 17:15:13 - deeznnutz: That purple screen.
Mon, Sep 9, 2024, 17:15:45 - coinselor: lol ya, that purple screen, I always press enter I don't even know what I'm looking at. Looks like some BIOS from 1980
Mon, Sep 9, 2024, 17:16:14 - deeznnutz: Exactly. I want to auto-select enter for those.
Mon, Sep 9, 2024, 17:16:20 - coinselor: But can that be skipped? Seems like some low-level VM stuff
Mon, Sep 9, 2024, 17:16:33 - deeznnutz: Yes, it can be auto-selected
Mon, Sep 9, 2024, 17:16:48 - deeznnutz: You tell it to accept all defaults and make the installation noninteractive.
Mon, Sep 9, 2024, 17:17:48 - georgezgeorgez: Noninteractive will be more important for multinode setups
Mon, Sep 9, 2024, 17:18:03 - georgezgeorgez: I'm guessing most people will still be SSH'ing to their box to install?
Mon, Sep 9, 2024, 17:18:16 - deeznnutz: ya, I think so
Mon, Sep 9, 2024, 17:18:36 - deeznnutz: I think we can target testing this week
Mon, Sep 9, 2024, 17:19:06 - deeznnutz: Improve readme.md
and noninteractive, and we can test I think
Mon, Sep 9, 2024, 17:19:09 - georgezgeorgez: Do we want to set a closer deadline to get some of these basics done? Then send out a call for beta testers?
Mon, Sep 9, 2024, 17:19:30 - deeznnutz: ya, I'd like to start testing by Friday this week
Mon, Sep 9, 2024, 17:21:04 - coinselor: My readme is chatgpt'd but I edited/proofread. I don't know what else could be added other than more content as we add more functionality
Mon, Sep 9, 2024, 17:21:19 - coinselor: It's on my forked repo tho
Mon, Sep 9, 2024, 17:21:31 - deeznnutz: Maybe add a tl;dr getting started?
Mon, Sep 9, 2024, 17:21:36 - georgezgeorgez: Make an MR
Mon, Sep 9, 2024, 17:22:22 - coinselor: OK, I will. Maybe 0x can edit and add stuff from his chatgpt readme XD
Mon, Sep 9, 2024, 17:24:20 - deeznnutz: Regarding next steps and roadmap, I typed up a few things. Maybe we should prioritize?
Mon, Sep 9, 2024, 17:24:23 - deeznnutz: Roadmap
- ARM support
- Automate the installation and setup of the
znnd
Dashboard - Troubleshooting script to send logs to devs
- Troubleshoot ports and accessibility (are my ports open)
- Deploy testnet
- Add
--uninstall
flag - Add
--backup
flag (I have a version of this script already)
Mon, Sep 9, 2024, 17:25:27 - deeznnutz: What is a higher priority? Automate the installation of the znnd
dash or troubleshooting "stuff"?
Mon, Sep 9, 2024, 17:25:57 - georgezgeorgez: I think we should prioritize a single happy path that is useful to most people first
Mon, Sep 9, 2024, 17:26:03 - coinselor: I can get a few aliens to commit to help us test it
Mon, Sep 9, 2024, 17:26:25 - georgezgeorgez: What would enable people to actually start using the script and running their nodes with it?
Mon, Sep 9, 2024, 17:26:43 - georgezgeorgez: The priority should be first to get some real users, and then after that, they can help us prioritize
Mon, Sep 9, 2024, 17:26:45 - deeznnutz: I like the znnd
dash. It's actually pretty useful
Mon, Sep 9, 2024, 17:27:14 - deeznnutz: So IMO the cool dash will be more useful to users than troubleshooting
Mon, Sep 9, 2024, 17:27:48 - georgezgeorgez: Yeah, I think we can get 1 and 2 to a good spot before focusing on 3
Mon, Sep 9, 2024, 17:27:52 - sugoi joined the room
Mon, Sep 9, 2024, 17:27:56 - georgezgeorgez: Usable spot
Mon, Sep 9, 2024, 17:28:07 - sugoi: Hi guys
Mon, Sep 9, 2024, 17:28:15 - georgezgeorgez: Hi, welcome to the operations SIG
Mon, Sep 9, 2024, 17:29:25 - georgezgeorgez: Not sure how much you have been following, but we've been working on a znnd
installation script and a dashboard to help people understand their node
Mon, Sep 9, 2024, 17:29:51 - deeznnutz: Does anyone have time to work on the auto-deploy of the znnd
dash? You can use some code already written in the script to deploy the dash w/ the API
Mon, Sep 9, 2024, 17:29:57 - coinselor: <@georgezgeorgez:hc1.chat "What would enable people to actu..."> I would personally start using it because it just automates a lot of the stuff. Grafana dashboard and other stuff will also come in handy if someone asks for system information.
Mon, Sep 9, 2024, 17:30:18 - sugoi: <@georgezgeorgezhc1.chat "not sure how much you have been ..."> Trying to read up now. Thanks for the tl;dr!
Mon, Sep 9, 2024, 17:30:39 - deeznnutz: So the dash requires:
- Install the infinity plugin
- Set the default datasource (127.0.0.1)
- Import the dashboard w/ the API
Mon, Sep 9, 2024, 17:30:49 - georgezgeorgez: Yes, we will likely be our own first users haha
Mon, Sep 9, 2024, 17:31:40 - georgezgeorgez: <@deeznnutz:zenon.chat "so the dash requires:"> Let's target this for Friday?
Mon, Sep 9, 2024, 17:31:41 - deeznnutz: It took me a while to figure out the import dash API. It had a bunch of size limits
Mon, Sep 9, 2024, 17:32:19 - deeznnutz: <@georgezgeorgez:hc1.chat "let's target this for friday?"> I can get the readme and noninteractive done by this Friday. Not sure about the znnd
dash
Mon, Sep 9, 2024, 17:32:28 - georgezgeorgez: I can work on it
Mon, Sep 9, 2024, 17:32:46 - deeznnutz: OK, if you have time, then I'm sure this Friday is possible.
Mon, Sep 9, 2024, 17:33:35 - deeznnutz: Just reuse this: https://github.com/go-zenon/go/blob/82ba64fbee715ad468c878a06cd16ac6258ccad1/grafana.sh#L119
Mon, Sep 9, 2024, 17:34:01 - georgezgeorgez: Yup.
0x, we can do it after this meeting and throughout the week, but let's get more GitHub issues created for all this
Mon, Sep 9, 2024, 17:34:11 - deeznnutz: That node exporter dash was very large and I kept getting file size errors
Mon, Sep 9, 2024, 17:34:15 - georgezgeorgez: I see you've created a couple already
Mon, Sep 9, 2024, 17:34:24 - coinselor: Yeah +1 to issues, break it down into tasks we can pick up
Mon, Sep 9, 2024, 17:34:38 - coinselor: For instance, I have no idea what this entails: Troubleshooting script to send logs to devs
Mon, Sep 9, 2024, 17:34:40 - deeznnutz: Yep. Makes sense
Mon, Sep 9, 2024, 17:35:25 - georgezgeorgez: Cool, so we have short-term goals for this week and also some other things on hold to revisit once we get it out for beta testing
Mon, Sep 9, 2024, 17:35:32 - georgezgeorgez: Anything else we want to go over?
Mon, Sep 9, 2024, 17:35:37 - georgezgeorgez: Or just ideas?
Mon, Sep 9, 2024, 17:35:58 - deeznnutz: Awesome. I can also work on adding/cleaning up the issues this week.
Mon, Sep 9, 2024, 17:36:01 - deeznnutz: That's all I had!
Mon, Sep 9, 2024, 17:36:26 - georgezgeorgez: Let's get another meeting on the cal
Mon, Sep 9, 2024, 17:36:32 - deeznnutz: I've gotten feedback from other devs watching this group and think we are making good progress on working together
Mon, Sep 9, 2024, 17:36:50 - deeznnutz: So I'm happy we are making these small steps together
Mon, Sep 9, 2024, 17:37:01 - coinselor: Just wishful thinking but would be badass to add --public
, --sentry
, --pillar
Mon, Sep 9, 2024, 17:37:01 - georgezgeorgez: I think even though it's all over chat, having regular times for the meetings helps moves things along
Mon, Sep 9, 2024, 17:37:09 - deeznnutz: 23 Sept @ 6PM EST?
Mon, Sep 9, 2024, 17:37:21 - georgezgeorgez: Message deleted
Mon, Sep 9, 2024, 17:37:33 - georgezgeorgez: <@coinselor:zenon.chat "just wishful thinking but would ..."> Likely on our path
Mon, Sep 9, 2024, 17:38:29 - georgezgeorgez: <@deeznnutz:zenon.chat "I've gotten feedback from other ..."> My goal with this SIG so far has been to bootstrap some sort of steady collaborative effort
Mon, Sep 9, 2024, 17:39:05 - coinselor: Particularly for the upcoming sporks, I know there's a way to like upgrade the pillar with no downtime. I'm not doing that unless this script can handle it 😂 no worth the effort, I rather miss the momentums lol
Mon, Sep 9, 2024, 17:39:21 - georgezgeorgez: Once we get some credibility, need to start figuring out incentives. Then the pace can pick up and really be driven by the community/userbase
Mon, Sep 9, 2024, 17:39:32 - georgezgeorgez: <@coinselor:zenon.chat "particularly for the upcoming sp..."> Bro, make an issue
Mon, Sep 9, 2024, 17:39:47 - georgezgeorgez: We need to track this stuff, even if it will be a bit before we get to it
Mon, Sep 9, 2024, 17:40:02 - georgezgeorgez: It helps too, because other people might chime in and say it's important for them too
Mon, Sep 9, 2024, 17:40:03 - deeznnutz: ya, we have a feature request tag in there
Mon, Sep 9, 2024, 17:40:12 - georgezgeorgez: And that will help us determine our priority
Mon, Sep 9, 2024, 17:40:35 - deeznnutz: Awesome. Is everyone OK with 23 Sept @ 6PM EST?
Mon, Sep 9, 2024, 17:40:39 - georgezgeorgez: But again, I'm pushing the saying:
Mon, Sep 9, 2024, 17:40:41 - georgezgeorgez: Incentivization is Prioritization
Mon, Sep 9, 2024, 17:41:05 - georgezgeorgez: My idea is that SIGs define the frontier of development. What is the next possible thing we can do?
Mon, Sep 9, 2024, 17:41:15 - georgezgeorgez: And the community helps prioritize it by setting incentives
Mon, Sep 9, 2024, 17:41:29 - deeznnutz: Makes perfect sense!
Mon, Sep 9, 2024, 17:41:51 - coinselor: OK, next step is to populate GitHub then
Mon, Sep 9, 2024, 17:41:53 - coinselor: Take it from there
Mon, Sep 9, 2024, 17:42:13 - deeznnutz: Cool. Thanks everyone. I'm around all week.
Mon, Sep 9, 2024, 17:42:22 - georgezgeorgez: Thank you everyone, good meeting
Mon, Sep 9, 2024, 17:42:29 - coinselor: gn!!
Mon, Sep 9, 2024, 17:42:33 - deeznnutz: gn