HC1: Operations SIG 23 Sept 2024: Difference between revisions
Updated minutes |
|||
(2 intermediate revisions by one other user not shown) | |||
Line 31: | Line 31: | ||
[[User:Coinselor|Coinselor]] | [[User:Coinselor|Coinselor]] | ||
* Created a forum poll to gather community feedback: https://forum.hypercore.one/t/community-poll-priority-enhacements-for-deployment-script/492 | |||
* Add <code>--deploy [options]</code> enhacement request https://github.com/hypercore-one/deployment/issues/19 | |||
== Meeting Minutes == | |||
'''Mon, Sep 23, 2024, 17:00:21 - deeznnutz''': === START OP SIG === | |||
'''Mon, Sep 23, 2024, 17:00:23 - deeznnutz''': Hello | |||
'''Mon, Sep 23, 2024, 17:00:43 - georgezgeorgez''': hello | |||
'''Mon, Sep 23, 2024, 17:01:56 - georgezgeorgez''': i guess to start I want to clarify why we have these meetings. We've been doing some good work async over chat. But the meetings help set a cadence and gives us some target dates for completing things. By posting the meeting minutes to the wiki, community members can follow along with just the high-level updates as well. | |||
'''Mon, Sep 23, 2024, 17:02:50 - georgezgeorgez''': We have some other SIGs starting up, such as #sig-syrius | |||
.chat, and some of the members here likely have to split our time between them and other initiatives. | |||
'''Mon, Sep 23, 2024, 17:03:27 - georgezgeorgez''': The SIG has agreed to move to a longer cadence, maybe every month or so. | |||
'''Mon, Sep 23, 2024, 17:03:51 - georgezgeorgez''': Now that we've released an initial product and will be relying on community feedback on next steps. | |||
'''Mon, Sep 23, 2024, 17:04:10 - georgezgeorgez''': We've also changed the scope of the SIG from Operations to Operations and Performance. | |||
'''Mon, Sep 23, 2024, 17:04:27 - georgezgeorgez''': Although the skillsets are different, the end outcome of both is the same. | |||
'''Mon, Sep 23, 2024, 17:04:33 - georgezgeorgez''': Community members being able to run infrastructure. | |||
'''Mon, Sep 23, 2024, 17:05:11 - georgezgeorgez''': Okay sorry, let's carry on. | |||
'''Mon, Sep 23, 2024, 17:05:18 - georgezgeorgez''': Just wanted to get some announcements out first. | |||
'''Mon, Sep 23, 2024, 17:05:27 - deeznnutz''': Good intro and thanks for that!! | |||
'''Mon, Sep 23, 2024, 17:05:45 - deeznnutz''': For those tuning in here is a quick summary. | |||
'''Mon, Sep 23, 2024, 17:05:52 - deeznnutz''': We accomplished our stated goals from the last meeting: | |||
* Gorg finished the <code>znnd</code> dashboard | |||
* We moved the repo to <code>/hypercore-one/deployment</code> and we released <code>v0.0.0-alpha</code> | |||
* The Community started to test the script and the feedback has been positive so far | |||
* We identified one bug. The <code>--start</code> flag did not work. We fixed that today. | |||
* We started to expand the Issues / Enhancement tracker on GitHub to track the roadmap | |||
* Coinselor posted on the forum looking for feedback on the most important features: <nowiki>https://forum.hypercore.one/t/community-poll-priority-enhacements-for-deployment-script/492</nowiki> | |||
'''Mon, Sep 23, 2024, 17:06:32 - deeznnutz''': I feel like we've made good progress on this small project and it's been useful on many levels. | |||
'''Mon, Sep 23, 2024, 17:06:37 - georgezgeorgez''': Great work everyone. | |||
'''Mon, Sep 23, 2024, 17:07:16 - georgezgeorgez''': I think the fact that other developers are joining these HC1 SIG channels is showing that this kind of collaboration model is working. | |||
'''Mon, Sep 23, 2024, 17:07:31 - deeznnutz''': ya I agree. | |||
'''Mon, Sep 23, 2024, 17:07:49 - coinselor''': I also enjoy the limited scopes; helps organizing efforts a lot. Good call. | |||
'''Mon, Sep 23, 2024, 17:07:57 - georgezgeorgez''': I really appreciate coinselor putting together that poll because it's exactly the next step we need to take. | |||
'''Mon, Sep 23, 2024, 17:08:10 - georgezgeorgez''': Having the community decide what the priorities are. | |||
'''Mon, Sep 23, 2024, 17:08:45 - georgezgeorgez''': People have expressed to me before how our development feels kind of trapped right now. Or that AZ is hostage to developer requests. | |||
'''Mon, Sep 23, 2024, 17:09:16 - georgezgeorgez''': We don't always get the value of a proposal immediately. But the community does not want to scare away our developers. | |||
'''Mon, Sep 23, 2024, 17:10:25 - georgezgeorgez''': So for these SIGs, I want us to move away from the model of large proposals priced in multiples of AZ. | |||
'''Mon, Sep 23, 2024, 17:10:42 - georgezgeorgez''': We can definitely break things down into small tasks, that the community can immediately get value from. | |||
'''Mon, Sep 23, 2024, 17:10:48 - georgezgeorgez''': And that don't cost a full AZ. | |||
'''Mon, Sep 23, 2024, 17:11:25 - deeznnutz''': makes sense to me. It also keeps the community informed and engaged in what devs are working on | |||
'''Mon, Sep 23, 2024, 17:11:51 - georgezgeorgez''': Maybe we can spend some time elaborating on each of the options in the poll? | |||
'''Mon, Sep 23, 2024, 17:12:04 - deeznnutz''': devs going away for a long time is stressful to community members I think | |||
'''Mon, Sep 23, 2024, 17:12:34 - georgezgeorgez''': yes and it discourages verification | |||
'''Mon, Sep 23, 2024, 17:12:49 - deeznnutz''': Regarding poll options: backup, I tried to summarize my thought here today: <nowiki>https://github.com/hypercore-one/deployment/issues/14</nowiki> | |||
Is this what we are thinking about or something more elaborate? | |||
'''Mon, Sep 23, 2024, 17:13:46 - georgezgeorgez''': Got it, so this item requires a bit of research first before implementation. | |||
'''Mon, Sep 23, 2024, 17:14:28 - coinselor''': <@georgezgeorgez.chat "I really appreciate coinselor putting together..."> On this note, I haven't spread the poll in other channels to see if we wanted to edit it / add more options from today. | |||
I also think we can clean up low-hanging fruits (like system checks) by next meeting and have a more 'stable' release we can publicly offer. | |||
'''Mon, Sep 23, 2024, 17:14:54 - georgezgeorgez''': I think it might also be good to limit the options | |||
'''Mon, Sep 23, 2024, 17:14:59 - georgezgeorgez''': Maybe to like 5? | |||
'''Mon, Sep 23, 2024, 17:15:28 - coinselor''': we can remove the 'hands-free' one, seems like that's the purpose to begin with anyway | |||
'''Mon, Sep 23, 2024, 17:15:35 - coinselor''': make it easier to run znnd | |||
'''Mon, Sep 23, 2024, 17:16:05 - georgezgeorgez''': Between meetings, we discuss what are likely the most valuable options. | |||
And pick the top few to put in a poll. | |||
We can put the others in a footnote or just on the wiki if people want to see what was left out but aren't following the meeting. | |||
'''Mon, Sep 23, 2024, 17:16:19 - georgezgeorgez''': That being said, I'm really glad to see some of our newer pillars so engaged in these meetings. | |||
'''Mon, Sep 23, 2024, 17:16:19 - deeznnutz''': <@coinselor.chat "we can remove the 'hands-free' one..."> was thinking the same. that is easy to update and we can probably do that while we wait for feedback. | |||
'''Mon, Sep 23, 2024, 17:17:09 - sugoi''': <@coinselor.chat "On this note, I haven't spread the poll..."> the poll is also 1 option only, maybe increase it to max 2? | |||
'''Mon, Sep 23, 2024, 17:17:13 - georgezgeorgez''': <@sugoi.chat "the poll is also 1 option only..."> I think that's fair. | |||
'''Mon, Sep 23, 2024, 17:18:08 - georgezgeorgez''': Yeah I agree also. Some of the ones, if they are basic, we can just do and notify. | |||
'''Mon, Sep 23, 2024, 17:18:41 - deeznnutz''': Are 1 & 3 sort of linked? | |||
'''Mon, Sep 23, 2024, 17:19:06 - georgezgeorgez''': I think 3 goes a bit deeper than 1 | |||
'''Mon, Sep 23, 2024, 17:19:23 - georgezgeorgez''': black box vs white box | |||
'''Mon, Sep 23, 2024, 17:19:57 - georgezgeorgez''': real-time monitoring of events would be the sync graphs to start I guess | |||
'''Mon, Sep 23, 2024, 17:20:03 - deeznnutz''': Should we call that simply, local backup (bootstrap)? | |||
'''Mon, Sep 23, 2024, 17:20:18 - georgezgeorgez''': and performance benchmarking gets a bit more specific, maybe testing specific momentum ranges | |||
'''Mon, Sep 23, 2024, 17:20:33 - coinselor''': <@georgezgeorgez.chat "While I hope most community members are able..."> I think to maximize community engagement later on, we could showcase the script capabilities with like a discord community call with video going over it. Maybe a Q&A. | |||
'''Mon, Sep 23, 2024, 17:20:47 - georgezgeorgez''': no longer about monitoring a running node, but specific tests for performance | |||
'''Mon, Sep 23, 2024, 17:21:38 - georgezgeorgez''': I do think we should have a mix of feedback channels. Maybe every month or even ad-hoc as needed, we post polls like you've done | |||
'''Mon, Sep 23, 2024, 17:21:53 - georgezgeorgez''': Maybe once a quarter or something we do a keynote event or something lmao | |||
'''Mon, Sep 23, 2024, 17:22:15 - georgezgeorgez''': and do demos | |||
'''Mon, Sep 23, 2024, 17:22:48 - georgezgeorgez''': We can definitely do one to kick off 2025, another zenon year | |||
'''Mon, Sep 23, 2024, 17:22:56 - georgezgeorgez''': do you think we can squeeze one in before then? | |||
'''Mon, Sep 23, 2024, 17:23:19 - deeznnutz''': sure, why not? | |||
'''Mon, Sep 23, 2024, 17:23:35 - coinselor''': seems plenty of time, even to add the pretty colors to the cli 😏 | |||
'''Mon, Sep 23, 2024, 17:23:45 - deeznnutz''': I'm curious, do we think this project can continue to live as a bunch of bash scripts? | |||
'''Mon, Sep 23, 2024, 17:23:53 - georgezgeorgez''': okay maybe let's do one in October. practice one. it's okay if it's a little scuffed | |||
'''Mon, Sep 23, 2024, 17:24:07 - georgezgeorgez''': well i think we've discussed a bit before about multi node orchestration | |||
'''Mon, Sep 23, 2024, 17:24:25 - deeznnutz''': ya, with ansible | |||
'''Mon, Sep 23, 2024, 17:24:25 - georgezgeorgez''': e.g. 1 pillar + 2 sentries | |||
'''Mon, Sep 23, 2024, 17:24:28 - coinselor''': <@deeznnutz.chat "I'm curious, do we think this project can..."> TON has a python wrapper, I don't think it's necessary but probably easier for developers. Bash not that friendly, I guess. | |||
'''Mon, Sep 23, 2024, 17:24:45 - georgezgeorgez''': yeah could be with ansible which is actually how most of us deployed these matrix servers i think | |||
'''Mon, Sep 23, 2024, 17:25:13 - deeznnutz''': <nowiki>https://docs.thorchain.org/thornodes/deploying</nowiki> | |||
'''Mon, Sep 23, 2024, 17:25:30 - deeznnutz''': eventually we will need a good set of docs / readme's | |||
'''Mon, Sep 23, 2024, 17:25:46 - georgezgeorgez''': i use kubernetes quite a bit for myself but i don't think we need to get to that level of complexity yet | |||
'''Mon, Sep 23, 2024, 17:26:04 - georgezgeorgez''': if we are at a point where we are dynamically provisioning new testnets on demand | |||
'''Mon, Sep 23, 2024, 17:26:13 - georgezgeorgez''': kubernetes might become useful to the wider community | |||
'''Mon, Sep 23, 2024, 17:26:28 - georgezgeorgez''': i think the next level of maturity would be something like ansible | |||
'''Mon, Sep 23, 2024, 17:26:53 - deeznnutz''': agree - really just showing off their docs and nice instructions <nowiki>https://docs.thorchain.org/thornodes/kubernetes/setup-linode</nowiki> | |||
'''Mon, Sep 23, 2024, 17:27:09 - georgezgeorgez''': yeah thorchain has a lot of operational complexity | |||
'''Mon, Sep 23, 2024, 17:27:15 - georgezgeorgez''': running a node of each bridged network | |||
'''Mon, Sep 23, 2024, 17:27:29 - georgezgeorgez''': so having kubernetes deployment be a widely supported default makes sense | |||
'''Mon, Sep 23, 2024, 17:28:11 - georgezgeorgez''': if someone is running znnd, an orchestrator, an extension chain node, a testnet node, and monitoring it all | |||
'''Mon, Sep 23, 2024, 17:28:17 - georgezgeorgez''': yeah it might be useful to us as well | |||
'''Mon, Sep 23, 2024, 17:28:19 - georgezgeorgez''': but let's build up to that | |||
'''Mon, Sep 23, 2024, 17:28:32 - deeznnutz''': in the poll, should we combine backup and restore into one activity? They really go together | |||
'''Mon, Sep 23, 2024, 17:28:43 - georgezgeorgez''': agree | |||
'''Mon, Sep 23, 2024, 17:28:46 - georgezgeorgez''': it's a single feature | |||
'''Mon, Sep 23, 2024, 17:29:06 - deeznnutz''': if we do that and remove the hands-free we are down to 5 | |||
'''Mon, Sep 23, 2024, 17:29:17 - georgezgeorgez''': nice | |||
'''Mon, Sep 23, 2024, 17:29:37 - georgezgeorgez''': i think if we present too much choice, it gets confusing | |||
'''Mon, Sep 23, 2024, 17:29:59 - georgezgeorgez''': long term, I would expect themes to emerge as well | |||
'''Mon, Sep 23, 2024, 17:30:13 - georgezgeorgez''': deployment, monitoring, resilience | |||
'''Mon, Sep 23, 2024, 17:30:22 - deeznnutz''': If we do go with backup / restore I have two very well tested scripts I've been using that we can leverage. | |||
'''Mon, Sep 23, 2024, 17:30:27 - georgezgeorgez''': and each of these will have a level of maturity | |||
'''Mon, Sep 23, 2024, 17:30:43 - georgezgeorgez''': the options could just be, which theme should we try to get to the next level of maturity | |||
'''Mon, Sep 23, 2024, 17:30:46 - georgezgeorgez''': which would include these things | |||
'''Mon, Sep 23, 2024, 17:31:05 - georgezgeorgez''': <@deeznnutz.chat "If we do go with backup / restore I have..."> great | |||
'''Mon, Sep 23, 2024, 17:31:50 - deeznnutz''': Sounds like next steps are to update the poll, send it out everywhere for feedback, and then allocate work based on the next features. | |||
'''Mon, Sep 23, 2024, 17:31:51 - georgezgeorgez''': We should really stress though that if you use someone else's backup without verifying it, you might as well not run your own node | |||
'''Mon, Sep 23, 2024, 17:32:14 - georgezgeorgez''': The whole point of running a node is to trustlessly verify the chain | |||
'''Mon, Sep 23, 2024, 17:32:21 - georgezgeorgez''': So taking your own backup and using it is great | |||
'''Mon, Sep 23, 2024, 17:32:28 - georgezgeorgez''': Using someone else's, not so great | |||
'''Mon, Sep 23, 2024, 17:32:55 - georgezgeorgez''': <@deeznnutz.chat "Sounds like next steps are to..."> yup. I'm curious to see what kind of feedback we get. | |||
'''Mon, Sep 23, 2024, 17:33:00 - deeznnutz''': <@georgezgeorgez.chat "You might as well not run your own node..."> agree. I think people do get frustrated with the sync times and some know I have a bootstrap and ask to use it. I try my hardest to avoid that, but some guys get frustrated | |||
'''Mon, Sep 23, 2024, 17:33:23 - georgezgeorgez''': I know we released the initial script, do we need to try and get more comms around it? | |||
'''Mon, Sep 23, 2024, 17:33:43 - deeznnutz''': I've been testing it offline with many people | |||
'''Mon, Sep 23, 2024, 17:33:52 - georgezgeorgez''': have we posted it in all the relevant channels or maybe even go for a tweet | |||
'''Mon, Sep 23, 2024, 17:33:56 - deeznnutz''': maybe 4 or 5 in addition to what we see in the open | |||
'''Mon, Sep 23, 2024, 17:33:57 - georgezgeorgez''': especially for the feedback | |||
'''Mon, Sep 23, 2024, 17:34:13 - georgezgeorgez''': i want people to know how we run things here | |||
'''Mon, Sep 23, 2024, 17:34:18 - deeznnutz''': Yes, let's try for more | |||
'''Mon, Sep 23, 2024, 17:34:31 - georgezgeorgez''': SIGs define work, Community prioritizes it. (with incentives, but that's WIP) | |||
'''Mon, Sep 23, 2024, 17:34:43 - coinselor''': sent an image. (Media omitted) | |||
'''Mon, Sep 23, 2024, 17:34:57 - georgezgeorgez''': <@deeznnutz.chat "maybe 4 or 5 in addition to what..."> Do they have any reservations about joining this group/chat? | |||
'''Mon, Sep 23, 2024, 17:35:04 - coinselor''': deleted the previous poll and set a new one to max 2 votes | |||
'''Mon, Sep 23, 2024, 17:35:12 - deeznnutz''': no I see many here | |||
'''Mon, Sep 23, 2024, 17:35:20 - deeznnutz''': Cap, Stark, Shai | |||
'''Mon, Sep 23, 2024, 17:35:48 - deeznnutz''': One more I'm forgetting | |||
'''Mon, Sep 23, 2024, 17:35:52 - georgezgeorgez''': Okay cool. Summary level is great and needed. But the more we can directly engage users, the better. | |||
'''Mon, Sep 23, 2024, 17:36:47 - georgezgeorgez''': Should we have cloud-specific guides? | |||
'''Mon, Sep 23, 2024, 17:37:02 - georgezgeorgez''': Like, how to deploy znnd on AWS? or how to deploy znnd on Digital Ocean? | |||
'''Mon, Sep 23, 2024, 17:37:03 - deeznnutz''': for now I don't think we need it | |||
'''Mon, Sep 23, 2024, 17:37:12 - deeznnutz''': it's pretty much the same | |||
'''Mon, Sep 23, 2024, 17:37:21 - georgezgeorgez''': Okay, the most relevant thing would be cloud firewalls | |||
'''Mon, Sep 23, 2024, 17:37:25 - deeznnutz''': only difference is if they don't have <code>apt</code> maybe | |||
'''Mon, Sep 23, 2024, 17:37:39 - deeznnutz''': <@georgezgeorgez.chat "Okay, the most relevant thing wo..."> ya, maybe we handle that in the readme | |||
'''Mon, Sep 23, 2024, 17:37:43 - georgezgeorgez''': well that's just ubuntu/debian based | |||
'''Mon, Sep 23, 2024, 17:37:50 - coinselor''': <@georgezgeorgez.chat "Okay, the most relevant thing wo..."> I think just having a few paragraphs noting the differences / explaining them would suffice | |||
'''Mon, Sep 23, 2024, 17:38:10 - georgezgeorgez''': Sounds good | |||
'''Mon, Sep 23, 2024, 17:38:44 - georgezgeorgez''': Cool, so we'll get that feedback out there. If people could try and share the initial deployment script and the poll that would be great. | |||
'''Mon, Sep 23, 2024, 17:39:00 - georgezgeorgez''': Anything else or should we set a time in about a month? | |||
'''Mon, Sep 23, 2024, 17:39:10 - deeznnutz''': And once we know what features people want we can break that work up | |||
'''Mon, Sep 23, 2024, 17:39:23 - coinselor''': <@deeznnutz.chat "only difference is if they don't..."> script should def check for this, either we don't run it if it's not ubuntu/debian or add support for others. I think it should be trivial to add arch/fedora/etc. The ton script had a check for it. | |||
'''Mon, Sep 23, 2024, 17:39:35 - georgezgeorgez''': Hmm, I wonder if we should mention Vilkris MR more explicitly in the poll | |||
'''Mon, Sep 23, 2024, 17:40:08 - georgezgeorgez''': Both are important, any tooling or process created as a result of getting those MRs through | |||
'''Mon, Sep 23, 2024, 17:40:13 - georgezgeorgez''': but just getting those MRs through also is good | |||
'''Mon, Sep 23, 2024, 17:40:33 - deeznnutz''': Side note... | |||
'''Mon, Sep 23, 2024, 17:40:36 - deeznnutz''': check out their status | |||
'''Mon, Sep 23, 2024, 17:40:39 - deeznnutz''': sent an image. (Media omitted) | |||
'''Mon, Sep 23, 2024, 17:40:47 - coinselor''': I think we can do that with the comms, meaning we share the poll and point to people being able to test pending MRs? | |||
'''Mon, Sep 23, 2024, 17:41:09 - georgezgeorgez''': sounds good. i just don't want it to get lost that we have those MRs | |||
'''Mon, Sep 23, 2024, 17:41:14 - deeznnutz''': I'm going to add that as an enhancement. let's not put on the poll. we can track separately | |||
'''Mon, Sep 23, 2024, 17:41:39 - georgezgeorgez''': <@deeznnutz.chat "sent an image."> haha well our ascii art is in good company | |||
'''Mon, Sep 23, 2024, 17:42:16 - georgezgeorgez''': yes, it might be cool to have a summary screen like this | |||
'''Mon, Sep 23, 2024, 17:42:26 - georgezgeorgez''': and then maybe a countdown that you can interrupt before it just takes off | |||
'''Mon, Sep 23, 2024, 17:43:09 - deeznnutz''': I think we have some good next steps. I have time this week to work on the hands-free changes. it should be pretty simple | |||
'''Mon, Sep 23, 2024, 17:43:33 - georgezgeorgez''': With a longer cadence, we should try and get our GitHub issues in a better spot. | |||
'''Mon, Sep 23, 2024, 17:43:35 - georgezgeorgez''': I'm guilty | |||
'''Mon, Sep 23, 2024, 17:43:43 - deeznnutz''': How does 21 Oct 24 @ 6PM EST work for the next meeting? | |||
'''Mon, Sep 23, 2024, 17:43:59 - georgezgeorgez''': and actually use them as part of our regular workflow | |||
'''Mon, Sep 23, 2024, 17:44:28 - georgezgeorgez''': mondays have been good for me so far | |||
'''Mon, Sep 23, 2024, 17:44:34 - deeznnutz''': cap: sugoi any other feedback you want to share? | |||
'''Mon, Sep 23, 2024, 17:45:32 - georgezgeorgez''': Hmm before we end, going back to incentives and community driving the work. | |||
'''Mon, Sep 23, 2024, 17:45:51 - sugoi''': <@deeznnutz.chat "cap: sugoi any other feedback you..."> I did not have the time yet to implement Vilkris' v2 thingy, but the script ran smooth. No issues there | |||
'''Mon, Sep 23, 2024, 17:47:05 - georgezgeorgez''': What I want to create here is like an engine with a little fire. We are going along right now keeping that fire alive and slowly chugging along. Each SIG is like an engine. | |||
The community's job is to decide which SIG to throw fuel (incentives) into so that we go much much faster. | |||
'''Mon, Sep 23, 2024, 17:47:15 - cap''': <@deeznnutz.chat "cap: sugoi any other feedback yo..."> nope, i think you guys are on the right track | |||
'''Mon, Sep 23, 2024, 17:47:47 - georgezgeorgez''': Wherever they put the fuel, we keep going. | |||
'''Mon, Sep 23, 2024, 17:48:10 - georgezgeorgez''': If Operations work is mostly stable, we still keep a small effort going but can focus on other SIGs. | |||
'''Mon, Sep 23, 2024, 17:48:21 - coinselor''': <@georgezgeorgez.chat "and actually use them as part of..."> I'll try to create smaller PRs for improvements to make the script more robust, tick off the easy stuff! | |||
'''Mon, Sep 23, 2024, 17:48:22 - georgezgeorgez''': At least, that's how I think it can work. | |||
'''Mon, Sep 23, 2024, 17:49:05 - georgezgeorgez''': Thank you sir. It all adds up. | |||
'''Mon, Sep 23, 2024, 17:49:43 - deeznnutz''': Great!! Glad we are humming along making progress. | |||
'''Mon, Sep 23, 2024, 17:50:45 - georgezgeorgez''': Over the next few weeks, we'll come up with some initial demo sessions. | |||
'''Mon, Sep 23, 2024, 17:50:54 - georgezgeorgez''': Thank you everyone, good meeting! | |||
'''Mon, Sep 23, 2024, 17:51:16 - deeznnutz''': I'll follow up on my items this week. Thanks everyone! | |||
'''Mon, Sep 23, 2024, 17:51:34 - coinselor''': I'm not even sure how deeznnutz figured out public nodes were offline, but I look forward to the day we have some bot leveraging logs or something we created to alert aliens in real time | |||
'''Mon, Sep 23, 2024, 17:52:16 - georgezgeorgez''': I think grafana can integrate with alerting services | |||
'''Mon, Sep 23, 2024, 17:52:27 - georgezgeorgez''': but let's continue that chat outside the meeting | |||
'''Mon, Sep 23, 2024, 17:52:37 - georgezgeorgez''': and properly end it for the minutes haha | |||
'''Mon, Sep 23, 2024, 17:53:03 - georgezgeorgez''': if people want details, they'll have to join | |||
'''Mon, Sep 23, 2024, 17:53:06 - coinselor''': Thanks for running it | |||
'''Mon, Sep 23, 2024, 17:53:08 - deeznnutz''': === END SIG === |
Latest revision as of 17:49, 27 September 2024
Agenda
What: Meeting to Discuss Improving Node Operations as part of the HC1: Operations SIG
When: 23 Sep 2024 @ 6PM EST
Where: https://matrix.to/#/#sig-operations:hc1.chat
Chair: 0x3639
Agenda:
- Discuss follow Up items from previous meeting
- Document action items
- Establish next meeting
If you want to attend please respond (or DM) with your full matrix username and I will invite you to the group. No FUD, anger or BS allowed.
Pre-meeting Notes
- Reviewed PRs and merged changes
- Closed open issues where complete
- Opened new issues and enhancement requests
-- start
needs to be fixed https://github.com/hypercore-one/deployment/issues/18- Moved and renamed repo to
/hypercore-one/deployment
- Add installation of znnd dashboard to grafana.sh https://github.com/go-zenon/go/pull/13
- Created a forum poll to gather community feedback: https://forum.hypercore.one/t/community-poll-priority-enhacements-for-deployment-script/492
- Add
--deploy [options]
enhacement request https://github.com/hypercore-one/deployment/issues/19
Meeting Minutes
Mon, Sep 23, 2024, 17:00:21 - deeznnutz: === START OP SIG ===
Mon, Sep 23, 2024, 17:00:23 - deeznnutz: Hello
Mon, Sep 23, 2024, 17:00:43 - georgezgeorgez: hello
Mon, Sep 23, 2024, 17:01:56 - georgezgeorgez: i guess to start I want to clarify why we have these meetings. We've been doing some good work async over chat. But the meetings help set a cadence and gives us some target dates for completing things. By posting the meeting minutes to the wiki, community members can follow along with just the high-level updates as well.
Mon, Sep 23, 2024, 17:02:50 - georgezgeorgez: We have some other SIGs starting up, such as #sig-syrius
.chat, and some of the members here likely have to split our time between them and other initiatives.
Mon, Sep 23, 2024, 17:03:27 - georgezgeorgez: The SIG has agreed to move to a longer cadence, maybe every month or so.
Mon, Sep 23, 2024, 17:03:51 - georgezgeorgez: Now that we've released an initial product and will be relying on community feedback on next steps.
Mon, Sep 23, 2024, 17:04:10 - georgezgeorgez: We've also changed the scope of the SIG from Operations to Operations and Performance.
Mon, Sep 23, 2024, 17:04:27 - georgezgeorgez: Although the skillsets are different, the end outcome of both is the same.
Mon, Sep 23, 2024, 17:04:33 - georgezgeorgez: Community members being able to run infrastructure.
Mon, Sep 23, 2024, 17:05:11 - georgezgeorgez: Okay sorry, let's carry on.
Mon, Sep 23, 2024, 17:05:18 - georgezgeorgez: Just wanted to get some announcements out first.
Mon, Sep 23, 2024, 17:05:27 - deeznnutz: Good intro and thanks for that!!
Mon, Sep 23, 2024, 17:05:45 - deeznnutz: For those tuning in here is a quick summary.
Mon, Sep 23, 2024, 17:05:52 - deeznnutz: We accomplished our stated goals from the last meeting:
- Gorg finished the
znnd
dashboard - We moved the repo to
/hypercore-one/deployment
and we releasedv0.0.0-alpha
- The Community started to test the script and the feedback has been positive so far
- We identified one bug. The
--start
flag did not work. We fixed that today. - We started to expand the Issues / Enhancement tracker on GitHub to track the roadmap
- Coinselor posted on the forum looking for feedback on the most important features: https://forum.hypercore.one/t/community-poll-priority-enhacements-for-deployment-script/492
Mon, Sep 23, 2024, 17:06:32 - deeznnutz: I feel like we've made good progress on this small project and it's been useful on many levels.
Mon, Sep 23, 2024, 17:06:37 - georgezgeorgez: Great work everyone.
Mon, Sep 23, 2024, 17:07:16 - georgezgeorgez: I think the fact that other developers are joining these HC1 SIG channels is showing that this kind of collaboration model is working.
Mon, Sep 23, 2024, 17:07:31 - deeznnutz: ya I agree.
Mon, Sep 23, 2024, 17:07:49 - coinselor: I also enjoy the limited scopes; helps organizing efforts a lot. Good call.
Mon, Sep 23, 2024, 17:07:57 - georgezgeorgez: I really appreciate coinselor putting together that poll because it's exactly the next step we need to take.
Mon, Sep 23, 2024, 17:08:10 - georgezgeorgez: Having the community decide what the priorities are.
Mon, Sep 23, 2024, 17:08:45 - georgezgeorgez: People have expressed to me before how our development feels kind of trapped right now. Or that AZ is hostage to developer requests.
Mon, Sep 23, 2024, 17:09:16 - georgezgeorgez: We don't always get the value of a proposal immediately. But the community does not want to scare away our developers.
Mon, Sep 23, 2024, 17:10:25 - georgezgeorgez: So for these SIGs, I want us to move away from the model of large proposals priced in multiples of AZ.
Mon, Sep 23, 2024, 17:10:42 - georgezgeorgez: We can definitely break things down into small tasks, that the community can immediately get value from.
Mon, Sep 23, 2024, 17:10:48 - georgezgeorgez: And that don't cost a full AZ.
Mon, Sep 23, 2024, 17:11:25 - deeznnutz: makes sense to me. It also keeps the community informed and engaged in what devs are working on
Mon, Sep 23, 2024, 17:11:51 - georgezgeorgez: Maybe we can spend some time elaborating on each of the options in the poll?
Mon, Sep 23, 2024, 17:12:04 - deeznnutz: devs going away for a long time is stressful to community members I think
Mon, Sep 23, 2024, 17:12:34 - georgezgeorgez: yes and it discourages verification
Mon, Sep 23, 2024, 17:12:49 - deeznnutz: Regarding poll options: backup, I tried to summarize my thought here today: https://github.com/hypercore-one/deployment/issues/14
Is this what we are thinking about or something more elaborate?
Mon, Sep 23, 2024, 17:13:46 - georgezgeorgez: Got it, so this item requires a bit of research first before implementation.
Mon, Sep 23, 2024, 17:14:28 - coinselor: <@georgezgeorgez.chat "I really appreciate coinselor putting together..."> On this note, I haven't spread the poll in other channels to see if we wanted to edit it / add more options from today.
I also think we can clean up low-hanging fruits (like system checks) by next meeting and have a more 'stable' release we can publicly offer.
Mon, Sep 23, 2024, 17:14:54 - georgezgeorgez: I think it might also be good to limit the options
Mon, Sep 23, 2024, 17:14:59 - georgezgeorgez: Maybe to like 5?
Mon, Sep 23, 2024, 17:15:28 - coinselor: we can remove the 'hands-free' one, seems like that's the purpose to begin with anyway
Mon, Sep 23, 2024, 17:15:35 - coinselor: make it easier to run znnd
Mon, Sep 23, 2024, 17:16:05 - georgezgeorgez: Between meetings, we discuss what are likely the most valuable options.
And pick the top few to put in a poll.
We can put the others in a footnote or just on the wiki if people want to see what was left out but aren't following the meeting.
Mon, Sep 23, 2024, 17:16:19 - georgezgeorgez: That being said, I'm really glad to see some of our newer pillars so engaged in these meetings.
Mon, Sep 23, 2024, 17:16:19 - deeznnutz: <@coinselor.chat "we can remove the 'hands-free' one..."> was thinking the same. that is easy to update and we can probably do that while we wait for feedback.
Mon, Sep 23, 2024, 17:17:09 - sugoi: <@coinselor.chat "On this note, I haven't spread the poll..."> the poll is also 1 option only, maybe increase it to max 2?
Mon, Sep 23, 2024, 17:17:13 - georgezgeorgez: <@sugoi.chat "the poll is also 1 option only..."> I think that's fair.
Mon, Sep 23, 2024, 17:18:08 - georgezgeorgez: Yeah I agree also. Some of the ones, if they are basic, we can just do and notify.
Mon, Sep 23, 2024, 17:18:41 - deeznnutz: Are 1 & 3 sort of linked?
Mon, Sep 23, 2024, 17:19:06 - georgezgeorgez: I think 3 goes a bit deeper than 1
Mon, Sep 23, 2024, 17:19:23 - georgezgeorgez: black box vs white box
Mon, Sep 23, 2024, 17:19:57 - georgezgeorgez: real-time monitoring of events would be the sync graphs to start I guess
Mon, Sep 23, 2024, 17:20:03 - deeznnutz: Should we call that simply, local backup (bootstrap)?
Mon, Sep 23, 2024, 17:20:18 - georgezgeorgez: and performance benchmarking gets a bit more specific, maybe testing specific momentum ranges
Mon, Sep 23, 2024, 17:20:33 - coinselor: <@georgezgeorgez.chat "While I hope most community members are able..."> I think to maximize community engagement later on, we could showcase the script capabilities with like a discord community call with video going over it. Maybe a Q&A.
Mon, Sep 23, 2024, 17:20:47 - georgezgeorgez: no longer about monitoring a running node, but specific tests for performance
Mon, Sep 23, 2024, 17:21:38 - georgezgeorgez: I do think we should have a mix of feedback channels. Maybe every month or even ad-hoc as needed, we post polls like you've done
Mon, Sep 23, 2024, 17:21:53 - georgezgeorgez: Maybe once a quarter or something we do a keynote event or something lmao
Mon, Sep 23, 2024, 17:22:15 - georgezgeorgez: and do demos
Mon, Sep 23, 2024, 17:22:48 - georgezgeorgez: We can definitely do one to kick off 2025, another zenon year
Mon, Sep 23, 2024, 17:22:56 - georgezgeorgez: do you think we can squeeze one in before then?
Mon, Sep 23, 2024, 17:23:19 - deeznnutz: sure, why not?
Mon, Sep 23, 2024, 17:23:35 - coinselor: seems plenty of time, even to add the pretty colors to the cli 😏
Mon, Sep 23, 2024, 17:23:45 - deeznnutz: I'm curious, do we think this project can continue to live as a bunch of bash scripts?
Mon, Sep 23, 2024, 17:23:53 - georgezgeorgez: okay maybe let's do one in October. practice one. it's okay if it's a little scuffed
Mon, Sep 23, 2024, 17:24:07 - georgezgeorgez: well i think we've discussed a bit before about multi node orchestration
Mon, Sep 23, 2024, 17:24:25 - deeznnutz: ya, with ansible
Mon, Sep 23, 2024, 17:24:25 - georgezgeorgez: e.g. 1 pillar + 2 sentries
Mon, Sep 23, 2024, 17:24:28 - coinselor: <@deeznnutz.chat "I'm curious, do we think this project can..."> TON has a python wrapper, I don't think it's necessary but probably easier for developers. Bash not that friendly, I guess.
Mon, Sep 23, 2024, 17:24:45 - georgezgeorgez: yeah could be with ansible which is actually how most of us deployed these matrix servers i think
Mon, Sep 23, 2024, 17:25:13 - deeznnutz: https://docs.thorchain.org/thornodes/deploying
Mon, Sep 23, 2024, 17:25:30 - deeznnutz: eventually we will need a good set of docs / readme's
Mon, Sep 23, 2024, 17:25:46 - georgezgeorgez: i use kubernetes quite a bit for myself but i don't think we need to get to that level of complexity yet
Mon, Sep 23, 2024, 17:26:04 - georgezgeorgez: if we are at a point where we are dynamically provisioning new testnets on demand
Mon, Sep 23, 2024, 17:26:13 - georgezgeorgez: kubernetes might become useful to the wider community
Mon, Sep 23, 2024, 17:26:28 - georgezgeorgez: i think the next level of maturity would be something like ansible
Mon, Sep 23, 2024, 17:26:53 - deeznnutz: agree - really just showing off their docs and nice instructions https://docs.thorchain.org/thornodes/kubernetes/setup-linode
Mon, Sep 23, 2024, 17:27:09 - georgezgeorgez: yeah thorchain has a lot of operational complexity
Mon, Sep 23, 2024, 17:27:15 - georgezgeorgez: running a node of each bridged network
Mon, Sep 23, 2024, 17:27:29 - georgezgeorgez: so having kubernetes deployment be a widely supported default makes sense
Mon, Sep 23, 2024, 17:28:11 - georgezgeorgez: if someone is running znnd, an orchestrator, an extension chain node, a testnet node, and monitoring it all
Mon, Sep 23, 2024, 17:28:17 - georgezgeorgez: yeah it might be useful to us as well
Mon, Sep 23, 2024, 17:28:19 - georgezgeorgez: but let's build up to that
Mon, Sep 23, 2024, 17:28:32 - deeznnutz: in the poll, should we combine backup and restore into one activity? They really go together
Mon, Sep 23, 2024, 17:28:43 - georgezgeorgez: agree
Mon, Sep 23, 2024, 17:28:46 - georgezgeorgez: it's a single feature
Mon, Sep 23, 2024, 17:29:06 - deeznnutz: if we do that and remove the hands-free we are down to 5
Mon, Sep 23, 2024, 17:29:17 - georgezgeorgez: nice
Mon, Sep 23, 2024, 17:29:37 - georgezgeorgez: i think if we present too much choice, it gets confusing
Mon, Sep 23, 2024, 17:29:59 - georgezgeorgez: long term, I would expect themes to emerge as well
Mon, Sep 23, 2024, 17:30:13 - georgezgeorgez: deployment, monitoring, resilience
Mon, Sep 23, 2024, 17:30:22 - deeznnutz: If we do go with backup / restore I have two very well tested scripts I've been using that we can leverage.
Mon, Sep 23, 2024, 17:30:27 - georgezgeorgez: and each of these will have a level of maturity
Mon, Sep 23, 2024, 17:30:43 - georgezgeorgez: the options could just be, which theme should we try to get to the next level of maturity
Mon, Sep 23, 2024, 17:30:46 - georgezgeorgez: which would include these things
Mon, Sep 23, 2024, 17:31:05 - georgezgeorgez: <@deeznnutz.chat "If we do go with backup / restore I have..."> great
Mon, Sep 23, 2024, 17:31:50 - deeznnutz: Sounds like next steps are to update the poll, send it out everywhere for feedback, and then allocate work based on the next features.
Mon, Sep 23, 2024, 17:31:51 - georgezgeorgez: We should really stress though that if you use someone else's backup without verifying it, you might as well not run your own node
Mon, Sep 23, 2024, 17:32:14 - georgezgeorgez: The whole point of running a node is to trustlessly verify the chain
Mon, Sep 23, 2024, 17:32:21 - georgezgeorgez: So taking your own backup and using it is great
Mon, Sep 23, 2024, 17:32:28 - georgezgeorgez: Using someone else's, not so great
Mon, Sep 23, 2024, 17:32:55 - georgezgeorgez: <@deeznnutz.chat "Sounds like next steps are to..."> yup. I'm curious to see what kind of feedback we get.
Mon, Sep 23, 2024, 17:33:00 - deeznnutz: <@georgezgeorgez.chat "You might as well not run your own node..."> agree. I think people do get frustrated with the sync times and some know I have a bootstrap and ask to use it. I try my hardest to avoid that, but some guys get frustrated
Mon, Sep 23, 2024, 17:33:23 - georgezgeorgez: I know we released the initial script, do we need to try and get more comms around it?
Mon, Sep 23, 2024, 17:33:43 - deeznnutz: I've been testing it offline with many people
Mon, Sep 23, 2024, 17:33:52 - georgezgeorgez: have we posted it in all the relevant channels or maybe even go for a tweet
Mon, Sep 23, 2024, 17:33:56 - deeznnutz: maybe 4 or 5 in addition to what we see in the open
Mon, Sep 23, 2024, 17:33:57 - georgezgeorgez: especially for the feedback
Mon, Sep 23, 2024, 17:34:13 - georgezgeorgez: i want people to know how we run things here
Mon, Sep 23, 2024, 17:34:18 - deeznnutz: Yes, let's try for more
Mon, Sep 23, 2024, 17:34:31 - georgezgeorgez: SIGs define work, Community prioritizes it. (with incentives, but that's WIP)
Mon, Sep 23, 2024, 17:34:43 - coinselor: sent an image. (Media omitted)
Mon, Sep 23, 2024, 17:34:57 - georgezgeorgez: <@deeznnutz.chat "maybe 4 or 5 in addition to what..."> Do they have any reservations about joining this group/chat?
Mon, Sep 23, 2024, 17:35:04 - coinselor: deleted the previous poll and set a new one to max 2 votes
Mon, Sep 23, 2024, 17:35:12 - deeznnutz: no I see many here
Mon, Sep 23, 2024, 17:35:20 - deeznnutz: Cap, Stark, Shai
Mon, Sep 23, 2024, 17:35:48 - deeznnutz: One more I'm forgetting
Mon, Sep 23, 2024, 17:35:52 - georgezgeorgez: Okay cool. Summary level is great and needed. But the more we can directly engage users, the better.
Mon, Sep 23, 2024, 17:36:47 - georgezgeorgez: Should we have cloud-specific guides?
Mon, Sep 23, 2024, 17:37:02 - georgezgeorgez: Like, how to deploy znnd on AWS? or how to deploy znnd on Digital Ocean?
Mon, Sep 23, 2024, 17:37:03 - deeznnutz: for now I don't think we need it
Mon, Sep 23, 2024, 17:37:12 - deeznnutz: it's pretty much the same
Mon, Sep 23, 2024, 17:37:21 - georgezgeorgez: Okay, the most relevant thing would be cloud firewalls
Mon, Sep 23, 2024, 17:37:25 - deeznnutz: only difference is if they don't have apt
maybe
Mon, Sep 23, 2024, 17:37:39 - deeznnutz: <@georgezgeorgez.chat "Okay, the most relevant thing wo..."> ya, maybe we handle that in the readme
Mon, Sep 23, 2024, 17:37:43 - georgezgeorgez: well that's just ubuntu/debian based
Mon, Sep 23, 2024, 17:37:50 - coinselor: <@georgezgeorgez.chat "Okay, the most relevant thing wo..."> I think just having a few paragraphs noting the differences / explaining them would suffice
Mon, Sep 23, 2024, 17:38:10 - georgezgeorgez: Sounds good
Mon, Sep 23, 2024, 17:38:44 - georgezgeorgez: Cool, so we'll get that feedback out there. If people could try and share the initial deployment script and the poll that would be great.
Mon, Sep 23, 2024, 17:39:00 - georgezgeorgez: Anything else or should we set a time in about a month?
Mon, Sep 23, 2024, 17:39:10 - deeznnutz: And once we know what features people want we can break that work up
Mon, Sep 23, 2024, 17:39:23 - coinselor: <@deeznnutz.chat "only difference is if they don't..."> script should def check for this, either we don't run it if it's not ubuntu/debian or add support for others. I think it should be trivial to add arch/fedora/etc. The ton script had a check for it.
Mon, Sep 23, 2024, 17:39:35 - georgezgeorgez: Hmm, I wonder if we should mention Vilkris MR more explicitly in the poll
Mon, Sep 23, 2024, 17:40:08 - georgezgeorgez: Both are important, any tooling or process created as a result of getting those MRs through
Mon, Sep 23, 2024, 17:40:13 - georgezgeorgez: but just getting those MRs through also is good
Mon, Sep 23, 2024, 17:40:33 - deeznnutz: Side note...
Mon, Sep 23, 2024, 17:40:36 - deeznnutz: check out their status
Mon, Sep 23, 2024, 17:40:39 - deeznnutz: sent an image. (Media omitted)
Mon, Sep 23, 2024, 17:40:47 - coinselor: I think we can do that with the comms, meaning we share the poll and point to people being able to test pending MRs?
Mon, Sep 23, 2024, 17:41:09 - georgezgeorgez: sounds good. i just don't want it to get lost that we have those MRs
Mon, Sep 23, 2024, 17:41:14 - deeznnutz: I'm going to add that as an enhancement. let's not put on the poll. we can track separately
Mon, Sep 23, 2024, 17:41:39 - georgezgeorgez: <@deeznnutz.chat "sent an image."> haha well our ascii art is in good company
Mon, Sep 23, 2024, 17:42:16 - georgezgeorgez: yes, it might be cool to have a summary screen like this
Mon, Sep 23, 2024, 17:42:26 - georgezgeorgez: and then maybe a countdown that you can interrupt before it just takes off
Mon, Sep 23, 2024, 17:43:09 - deeznnutz: I think we have some good next steps. I have time this week to work on the hands-free changes. it should be pretty simple
Mon, Sep 23, 2024, 17:43:33 - georgezgeorgez: With a longer cadence, we should try and get our GitHub issues in a better spot.
Mon, Sep 23, 2024, 17:43:35 - georgezgeorgez: I'm guilty
Mon, Sep 23, 2024, 17:43:43 - deeznnutz: How does 21 Oct 24 @ 6PM EST work for the next meeting?
Mon, Sep 23, 2024, 17:43:59 - georgezgeorgez: and actually use them as part of our regular workflow
Mon, Sep 23, 2024, 17:44:28 - georgezgeorgez: mondays have been good for me so far
Mon, Sep 23, 2024, 17:44:34 - deeznnutz: cap: sugoi any other feedback you want to share?
Mon, Sep 23, 2024, 17:45:32 - georgezgeorgez: Hmm before we end, going back to incentives and community driving the work.
Mon, Sep 23, 2024, 17:45:51 - sugoi: <@deeznnutz.chat "cap: sugoi any other feedback you..."> I did not have the time yet to implement Vilkris' v2 thingy, but the script ran smooth. No issues there
Mon, Sep 23, 2024, 17:47:05 - georgezgeorgez: What I want to create here is like an engine with a little fire. We are going along right now keeping that fire alive and slowly chugging along. Each SIG is like an engine.
The community's job is to decide which SIG to throw fuel (incentives) into so that we go much much faster.
Mon, Sep 23, 2024, 17:47:15 - cap: <@deeznnutz.chat "cap: sugoi any other feedback yo..."> nope, i think you guys are on the right track
Mon, Sep 23, 2024, 17:47:47 - georgezgeorgez: Wherever they put the fuel, we keep going.
Mon, Sep 23, 2024, 17:48:10 - georgezgeorgez: If Operations work is mostly stable, we still keep a small effort going but can focus on other SIGs.
Mon, Sep 23, 2024, 17:48:21 - coinselor: <@georgezgeorgez.chat "and actually use them as part of..."> I'll try to create smaller PRs for improvements to make the script more robust, tick off the easy stuff!
Mon, Sep 23, 2024, 17:48:22 - georgezgeorgez: At least, that's how I think it can work.
Mon, Sep 23, 2024, 17:49:05 - georgezgeorgez: Thank you sir. It all adds up.
Mon, Sep 23, 2024, 17:49:43 - deeznnutz: Great!! Glad we are humming along making progress.
Mon, Sep 23, 2024, 17:50:45 - georgezgeorgez: Over the next few weeks, we'll come up with some initial demo sessions.
Mon, Sep 23, 2024, 17:50:54 - georgezgeorgez: Thank you everyone, good meeting!
Mon, Sep 23, 2024, 17:51:16 - deeznnutz: I'll follow up on my items this week. Thanks everyone!
Mon, Sep 23, 2024, 17:51:34 - coinselor: I'm not even sure how deeznnutz figured out public nodes were offline, but I look forward to the day we have some bot leveraging logs or something we created to alert aliens in real time
Mon, Sep 23, 2024, 17:52:16 - georgezgeorgez: I think grafana can integrate with alerting services
Mon, Sep 23, 2024, 17:52:27 - georgezgeorgez: but let's continue that chat outside the meeting
Mon, Sep 23, 2024, 17:52:37 - georgezgeorgez: and properly end it for the minutes haha
Mon, Sep 23, 2024, 17:53:03 - georgezgeorgez: if people want details, they'll have to join
Mon, Sep 23, 2024, 17:53:06 - coinselor: Thanks for running it
Mon, Sep 23, 2024, 17:53:08 - deeznnutz: === END SIG ===