Operations SIG 22 July 2024
Agenda
What: Meeting to Discuss Improving Node Operations
When: 22 July 2024 @ 6PM EST
Where: https://matrix.to/#/#sig-operations:hc1.chat
Agenda:
- Discuss purpose of the group
- Establish general goals of the group
- Discuss meeting cadence and deliverables
If you want to attend please respond (or DM) with your full matrix username and I will invite you to the group. No FUD, anger or BS allowed.
Minutes
Mon, Jul 22, 2024, 17:00:44 - georgezgeorgez: Okay it's 6pm EST. We can get started.
Mon, Jul 22, 2024, 17:01:14 - georgezgeorgez: Thank you everyone who is attending.
This is the first meeting of the HC1 Operations Special Interest Group (SIG).
Mon, Jul 22, 2024, 17:01:25 - georgezgeorgez: First I'll describe some motivation.
Mon, Jul 22, 2024, 17:01:47 - georgezgeorgez: I think we're at a point now where community members can see progress but it is very vague.
Mon, Jul 22, 2024, 17:02:02 - georgezgeorgez: There are also some really cool big features being worked on.
Mon, Jul 22, 2024, 17:02:38 - georgezgeorgez: There is an adage of the 80/20 rule. Where the last 20% of the work takes 80% of the effort.
Mon, Jul 22, 2024, 17:03:11 - georgezgeorgez: In terms of software delivery, that last 20% gives us the confidence to actually release the software to users in production.
Mon, Jul 22, 2024, 17:03:45 - georgezgeorgez: That last 20% consists of things such as testing, documentation, etc.
Mon, Jul 22, 2024, 17:04:56 - georgezgeorgez: We always appreciate the contributors who are able to deliver fully end to end.
Mon, Jul 22, 2024, 17:05:35 - georgezgeorgez: But if we want to be a welcoming project that attracts many contributors, we will likely need to provide some facilitation.
Mon, Jul 22, 2024, 17:05:45 - georgezgeorgez: So that's what the HC1 SIGs are for.
Mon, Jul 22, 2024, 17:06:06 - georgezgeorgez: I wanted to start with Operations first because it is somewhat of a dependency for all future work we do.
Mon, Jul 22, 2024, 17:07:05 - georgezgeorgez: Being able to spin up infrastructure and monitor and debug it, is not only crucial for our users, but also for contributors.
Mon, Jul 22, 2024, 17:07:35 - georgezgeorgez: Faster deployments mean the ability to quickly create testnets and faster iterations cycles.
Mon, Jul 22, 2024, 17:08:12 - georgezgeorgez: As our testing maturity improves, the operational tools that testing relies on will need to improve as well.
Mon, Jul 22, 2024, 17:08:42 - georgezgeorgez: I've asked 0x to help chair the SIG because I know this is an area of interest and personal experience for him.
Mon, Jul 22, 2024, 17:09:40 - georgezgeorgez: In additional to his own experience, I've asked him to work with our community/users to help get work defined when it comes to Operations.
Mon, Jul 22, 2024, 17:10:56 - georgezgeorgez: With well defined work, HC1 and other contributors (maybe even some new ones), can help tackle many of the smaller but critical contributions needed.
Mon, Jul 22, 2024, 17:12:40 - georgezgeorgez: I'd also like for the community to consider incentivizing this chair position. Project management and work definition is not work that everyone wants to do, but again a necessary labor.
Mon, Jul 22, 2024, 17:12:50 - georgezgeorgez: With that said, I'll hand it over to 0x.
Mon, Jul 22, 2024, 17:12:58 - georgezgeorgez: As chair, he will run the SIG as he sees fit.
Mon, Jul 22, 2024, 17:13:14 - deeznnutz (@deeznnutz:zenon.chat): ty sir. I have some copy / paste stuff, some is overlap
Mon, Jul 22, 2024, 17:13:18 - deeznnutz (@deeznnutz:zenon.chat): so please excuse that
Mon, Jul 22, 2024, 17:13:26 - georgezgeorgez: yeah sorry, I am slow and should have prepared that beforehand
Mon, Jul 22, 2024, 17:13:33 - deeznnutz (@deeznnutz:zenon.chat): NP
Mon, Jul 22, 2024, 17:13:46 - deeznnutz (@deeznnutz:zenon.chat): Before we get started, let's review the chat rules:
1) Disagreements are expected, but be respectful of others
2) Don't be a dick
3) No FUD
Any violation of these rules will result in an immediate and permanent ban.  
Any questions / comments?
Mon, Jul 22, 2024, 17:14:04 - georgezgeorgez: Makes sense.
Mon, Jul 22, 2024, 17:14:27 - deeznnutz (@deeznnutz:zenon.chat): **Introduction**
George and I have been discussing establishing Special Interest Groups (SIG) to enable community members to focus their skills on areas of interest within Zenon Network. Given my engineering background and experience supporting pillars, node operators, and orchestrators I wanted to start up the first Special Interest Group that focuses on Network Operations.  
We hope other community members will take an interest and start their own Special Interest Groups within Zenon. A Hypergrowth (Marketing) SIG is an example of a group other community members could start and run. Stark, Brat, Sugoi, Zyler and others seem uniquely qualified to run a grass roots marketing SIG.  
We plan to report on and track the progress of this Operations SIG here: <https://zenon.wiki/index.php/HC1:_Operations_SIG>
Any questions or comments?
Mon, Jul 22, 2024, 17:15:57 - georgezgeorgez: I think it's okay to go ahead. Matrix has threading features. And we will export the chat as well.
Mon, Jul 22, 2024, 17:16:03 - deeznnutz (@deeznnutz:zenon.chat): cool
Mon, Jul 22, 2024, 17:16:07 - deeznnutz (@deeznnutz:zenon.chat): **Operations SIG Purpose:**
The purpose of this **Operations SIG** is to discuss, improve and support the operations of:
1) go-zenon
2) orchestrators
3) Supernova Validators
4) UnikernelZ
5) Sentinels (TBD)
6) Sentries (TBD)
Once we finalize the scope of the group and establish deliverables we expect to submit an AZ to support the group and its goals. Our hope is we can show how AZ can be used to help grow and improve the network without needing to write code in go, ocamel or .net.  
Does anyone recommend any changes or additions to the purpose of the group?  
Mon, Jul 22, 2024, 17:16:45 - kenkef: what is AZ
Mon, Jul 22, 2024, 17:17:07 - deeznnutz (@deeznnutz:zenon.chat): Is the on chain funding framework
Mon, Jul 22, 2024, 17:18:04 - georgezgeorgez: That's a fair amount of scope. I would suggest to start small, like the go-zenon deployer cli, and first get that in a good spot.
Mon, Jul 22, 2024, 17:18:31 - deeznnutz (@deeznnutz:zenon.chat): yes, that was my thinking too.
Mon, Jul 22, 2024, 17:18:44 - deeznnutz (@deeznnutz:zenon.chat): Start there, see how it goes, and expand
Mon, Jul 22, 2024, 17:19:08 - deeznnutz (@deeznnutz:zenon.chat): <@kenkef:matrix.org "what is AZ"> https://medium.com/@Zyler9985/simple-guideline-for-az-proposals-94621c80f257
Mon, Jul 22, 2024, 17:19:30 - georgezgeorgez: What we need to start practicing is defining incremental pieces of work that others can easily pick up and complete, say within a week or so.
Mon, Jul 22, 2024, 17:19:58 - deeznnutz (@deeznnutz:zenon.chat): how should we handle things like research?
Mon, Jul 22, 2024, 17:21:11 - georgezgeorgez: For open ended things like that, I think what is important is to make sure the questions are well defined. What questions is the research trying to answer.
Mon, Jul 22, 2024, 17:21:55 - georgezgeorgez: We will need to figure out how to define the incentives for both well defined pieces of work (e.g change this function) as well as more vague ones.
Mon, Jul 22, 2024, 17:21:57 - coinselor joined the room
Mon, Jul 22, 2024, 17:22:00 - georgezgeorgez: But we'll get there later.
Mon, Jul 22, 2024, 17:22:11 - georgezgeorgez: Let's start with just getting good at defining the work in small increments.
Mon, Jul 22, 2024, 17:22:23 - deeznnutz (@deeznnutz:zenon.chat): ok - this ties into my proposed first goal
Mon, Jul 22, 2024, 17:22:31 - deeznnutz (@deeznnutz:zenon.chat): **Group Goals:**
The proposed first goal of the group is create a "framework" to launch, monitor, backup, restore and generally operate `go-zenon` as easily as possible. The goal is a "one click" launch so it's very easy to spin up, monitor and manage infrastructure. Once this framework is developed we can apply it to other nodes in the network.  
Example frameworks could include:
* docker compose stack (test implementation done)
* Example <https://github.com/0x3639/znndNode>
* Bash script(s) (test implementation WIP)
* Example (WIP) <https://gist.github.com/0x3639/979e2b0d2b0acea31dfd30849a524d4a>
* Expanding znn-controller
* Ansible
* Get Umbrel <https://github.com/getumbrel/umbrel>
* Other
Open Source Monitoring & Reporting Tools
* Monit - <https://mmonit.com/monit/>
* Grafana - <https://grafana.com/>
Any comments on the group's first goal and/or example frameworks we should consider?
Mon, Jul 22, 2024, 17:23:29 - georgezgeorgez: No, I think that is great.
I like too that you provide different options.
Mon, Jul 22, 2024, 17:23:50 - georgezgeorgez: As the SIG chair, you need to "love the problem" and not any particular solution.
Mon, Jul 22, 2024, 17:24:26 - kenkef: <@deeznnutz:zenon.chat "**Group Goals:**"> so these first two examples are works in progress and the goal would be to create alternatives to those?
Mon, Jul 22, 2024, 17:24:48 - deeznnutz (@deeznnutz:zenon.chat): I think the goal is to create one solution and stick with it
Mon, Jul 22, 2024, 17:25:15 - deeznnutz (@deeznnutz:zenon.chat): before we discussed this group I've mostly been messing around with solutions with no real goal in mind
Mon, Jul 22, 2024, 17:25:37 - kenkef: Why not use the docker one if it already works?
Mon, Jul 22, 2024, 17:26:32 - deeznnutz (@deeznnutz:zenon.chat): yes that is possible. we would need to improve it. I've test deployed this with CF and it's not so easy. We need to make it much easier. One-click style
Mon, Jul 22, 2024, 17:27:03 - deeznnutz (@deeznnutz:zenon.chat): The bash script is much closer to one-click but I have not integrated monitoring yet.
Mon, Jul 22, 2024, 17:27:50 - deeznnutz (@deeznnutz:zenon.chat): I think we can make the docker implementaion easier for sure
Mon, Jul 22, 2024, 17:28:57 - georgezgeorgez: <@kenkef:matrix.org "Why not use the docker one if it..."> I think this is a good question that demonstrates what we're trying to do here.
In the community, we have things that "work". But most of them are not being actively maintained or developed.
Mon, Jul 22, 2024, 17:29:34 - georgezgeorgez: There's a good amount of things like keeping dependencies up to date that no one is doing, or really has any incentive to do.
Mon, Jul 22, 2024, 17:30:19 - deeznnutz (@deeznnutz:zenon.chat): I did spend a lot of time setting up the dashboards and deployment of grafana. It's automatic and works great. it spins up all the dashboards autoamtically
Mon, Jul 22, 2024, 17:30:54 - deeznnutz (@deeznnutz:zenon.chat): In terms of speed, I could get a working deployment on docker faster than a bash script I think
Mon, Jul 22, 2024, 17:31:28 - deeznnutz (@deeznnutz:zenon.chat): I also have a good set of instructions to deploy a testnet and could easily modify the docker compose to do that too.
Mon, Jul 22, 2024, 17:34:40 - coinselor: here's some noob thoughts from having deployed and ran a few instances of go-zenon, some public:
- the biggest pain is setting up the env, a bash script obviously helps but I believe this has limitations / doesn't work on all server OSes/envs. Couldn't a beefier znn-controller act as "cli-ui" for a more powerful/customizable bash script?
- which is why people use containers, I guess. Have very little experience with docker and had some issues with persistent data when I did use it, but I guess this can be easier to deploy with right instructions
- never used getumbrel but I believe a UI friendly solution like that can be powerful if we target non techies
Mon, Jul 22, 2024, 17:35:54 - kenkef: why will non-techies need a wrapper for go-zenon won't they use the syrius wallet?
Mon, Jul 22, 2024, 17:36:30 - georgezgeorgez: So looking at https://github.com/getumbrel/umbrel
> A beautiful home server OS for self-hosting with an app store. Buy a pre-built Umbrel Home with umbrelOS, or install on a Raspberry Pi or any x86 system.
I think this target audience is too narrow for us to start with.
Mon, Jul 22, 2024, 17:37:30 - georgezgeorgez: <@kenkef:matrix.org "why will non-techies need a wrap..."> There's a good portion of our pillar operators who are probably technical enough to follow instructions but not understand underlying details.
Mon, Jul 22, 2024, 17:37:59 - deeznnutz (@deeznnutz:zenon.chat): <@kenkef:matrix.org "why will non-techies need a wrap..."> Yes I think that is the case for normal users. I think we are targeting people who want to run their own node to support the network and/or a dev who wants to spin up a testnet quickly (in a few clicks) to start working on the network. WHen I tested CFs PTLCs it took me about 1 hours from start to finish to setup the testnet.
Mon, Jul 22, 2024, 17:38:52 - georgezgeorgez: > Don't trust. Verify.
Let's lower the barrier of verification.
Mon, Jul 22, 2024, 17:40:22 - deeznnutz (@deeznnutz:zenon.chat): Regarding Brat's comments, we can overcome building the env and the scripts work on mac and linux. I think integrating grafana will be harder than using docker images. Also, the docker compose I setup also integrated caddy if someone wants to host a public node
Mon, Jul 22, 2024, 17:40:27 - deeznnutz (@deeznnutz:zenon.chat): so there is a lot in there
Mon, Jul 22, 2024, 17:40:47 - deeznnutz (@deeznnutz:zenon.chat): and it's pretty much automatic after providing some variables. We just need to make it even easier to deploy.
Mon, Jul 22, 2024, 17:41:07 - georgezgeorgez: 0x, so I think a good goal would be: to have 2 or 3 tickets starting to take form by the next meeting or so. Doesn't have to be fully defined, but just directional.
Mon, Jul 22, 2024, 17:42:13 - georgezgeorgez: In the future, I think we will work on some community voting, weighing systems to help pick which tickets to prioritize or choose over the others. But we are just starting now.
Mon, Jul 22, 2024, 17:42:39 - deeznnutz (@deeznnutz:zenon.chat): Cool. I can work on that for sure. How often does everyone want to meet?
Mon, Jul 22, 2024, 17:43:43 - deeznnutz (@deeznnutz:zenon.chat): We can do every 2 or 4 weeks and we can chat in here in between meetings.
Mon, Jul 22, 2024, 17:44:11 - coinselor: Just use common sense imho. Whatever makes your life easier when deploying stuff will probably make everyone's life easier. We just need to clean up / document the solutions provided. Maybe offer some flexibility: more/less monitoring, etc
Mon, Jul 22, 2024, 17:45:24 - deeznnutz (@deeznnutz:zenon.chat): I can focus my energy on the docker stack. I can identify the issues with what we have today and propose ways to improve.
Mon, Jul 22, 2024, 17:46:06 - georgezgeorgez: <@deeznnutz:zenon.chat "We can do every 2 or 4 weeks and..."> Hmm, let's do the next one sooner in 2. And once we have a cadence, we can adjust.
Mon, Jul 22, 2024, 17:46:36 - georgezgeorgez: There's no work defined that others can pick up yet.
Mon, Jul 22, 2024, 17:47:13 - deeznnutz (@deeznnutz:zenon.chat): maybe we can flush that out after I get back into the docker compose file
Mon, Jul 22, 2024, 17:47:18 - deeznnutz (@deeznnutz:zenon.chat): it's been a while
Mon, Jul 22, 2024, 17:48:12 - deeznnutz (@deeznnutz:zenon.chat): Maybe I can get it running in it's current form. coinselor can test and we can figure out the weakness / areas of improvement?
Mon, Jul 22, 2024, 17:50:24 - georgezgeorgez: <@deeznnutz:zenon.chat "I can focus my energy on the doc..."> Again let's start small. If there are some tickets for work that can be done in an hour or afternoon, let's get those defined first.
Mon, Jul 22, 2024, 17:50:58 - georgezgeorgez: Longer term, it'll be good to have a mix of complexity, for newer contributors and more experienced ones.
Mon, Jul 22, 2024, 17:51:18 - georgezgeorgez: And of course, if the work is already well defined, just go into engineering mode and start knocking them out.
Mon, Jul 22, 2024, 17:51:24 - deeznnutz (@deeznnutz:zenon.chat): <@georgezgeorgez:hc1.chat "Again let's start small. If ther..."> when you say tickets, are you thinking we create issues in GH to be assigned?
Mon, Jul 22, 2024, 17:51:40 - georgezgeorgez: Up to you, whatever is best for the SIG.
Mon, Jul 22, 2024, 17:52:40 - deeznnutz (@deeznnutz:zenon.chat): I'll start by getting back into the project first and assess the situation.
Mon, Jul 22, 2024, 17:52:57 - georgezgeorgez: I want to keep the SIGs fairly autonomous and not put too many requirements.
Whatever is the most productive for their mission.
Engineers from different problem domains, likely have different normal workflows and tooling.
Mon, Jul 22, 2024, 17:53:26 - deeznnutz (@deeznnutz:zenon.chat): Cool - works for me.
Mon, Jul 22, 2024, 17:53:34 - coinselor: Happy to test. Btw for actual problems/solutions:
- I've never load balance/setup caddy for the public node. Been procastinating on that.
- I've never setup a monitoring solution for go-zenon. I usually just reboot when I notice they are down.
- Sentrifying is a pain in the ass, sultan's tutorials helped.
Mon, Jul 22, 2024, 17:54:08 - georgezgeorgez: Cool, you can be our target user to start.
Mon, Jul 22, 2024, 17:54:25 - deeznnutz (@deeznnutz:zenon.chat): Ya I think that is perfect!!
Mon, Jul 22, 2024, 17:54:42 - deeznnutz (@deeznnutz:zenon.chat): Anything else before we break? Next meeting 5 Aug at 6PM EST? Does that work?
Mon, Jul 22, 2024, 17:55:03 - georgezgeorgez: Monday, should work for me.
Mon, Jul 22, 2024, 17:55:04 - deeznnutz (@deeznnutz:zenon.chat): I will post the minutes in zenon.wiki and announce the next meeting in the forum
Mon, Jul 22, 2024, 17:55:51 - georgezgeorgez: Thank you 0x! For chairing the SIG and a great first meeting.
Mon, Jul 22, 2024, 17:56:11 - deeznnutz (@deeznnutz:zenon.chat): thx everyone!! see you and your alts in TG!!
Mon, Jul 22, 2024, 17:56:23 - coinselor: Make the public announcement for next one ahead of time 😅
Mon, Jul 22, 2024, 17:56:29 - deeznnutz (@deeznnutz:zenon.chat): yes!!!
Mon, Jul 22, 2024, 17:56:36 - georgezgeorgez: Let's keep up the Momentum(tm)
Mon, Jul 22, 2024, 17:57:00 - edgepillar: automate Zenon ⚙️