Operations SIG 03 March 2025: Difference between revisions
update george notes |
No edit summary |
||
Line 26: | Line 26: | ||
[[User:Vilkris|Vilkris]] | [[User:Vilkris|Vilkris]] | ||
== Meeting | == Meeting Minutes Summary (chatGPT) == | ||
== Meeting Minutes == | |||
Mon, Mar 3, 2025, 11:59:57 - deeznnutz: ==== OG SIG Start ==== | |||
Mon, Mar 3, 2025, 12:00:17 - deeznnutz: GM. I dont have anything prepared to start with, but did make progress on ansible | |||
Mon, Mar 3, 2025, 12:00:36 - deeznnutz: It easy to see how this will be very helpful and way more powerful than the scripts we were using before | |||
Mon, Mar 3, 2025, 12:00:55 - deeznnutz: georgezgeorgez: did you take and look and any initial feedback? Was I using Roles correctly? | |||
Mon, Mar 3, 2025, 12:01:17 - georgezgeorgez: i took an initial look but haven't gone through everything yet | |||
Mon, Mar 3, 2025, 12:01:24 - georgezgeorgez: i think what's there is good | |||
Mon, Mar 3, 2025, 12:01:33 - georgezgeorgez: i might make some PRs in the coming weeks | |||
Mon, Mar 3, 2025, 12:01:41 - georgezgeorgez: but otherwise keep it up | |||
Mon, Mar 3, 2025, 12:01:50 - georgezgeorgez: there are some concepts like "handlers" | |||
Mon, Mar 3, 2025, 12:01:59 - georgezgeorgez: that we can take advantage as well | |||
Mon, Mar 3, 2025, 12:02:11 - georgezgeorgez: when refamiliarizing myself with ansible last week | |||
Mon, Mar 3, 2025, 12:02:23 - georgezgeorgez: i think one tricky part is actually not part of these playbooks | |||
Mon, Mar 3, 2025, 12:02:30 - georgezgeorgez: it's the setting up of the ssh connection | |||
Mon, Mar 3, 2025, 12:02:36 - deeznnutz: lol | |||
Mon, Mar 3, 2025, 12:02:38 - deeznnutz: yes!! | |||
Mon, Mar 3, 2025, 12:02:47 - georgezgeorgez: Ansible prefers ssh keypairs | |||
Mon, Mar 3, 2025, 12:03:10 - georgezgeorgez: but i'm guessing a lot of people are just using passwords | |||
Mon, Mar 3, 2025, 12:03:18 - deeznnutz: yes I was going to mention that. I used `--ask-pass` to get around that but it requires local packages to use that flag | |||
Mon, Mar 3, 2025, 12:03:26 - georgezgeorgez: yes exactly | |||
Mon, Mar 3, 2025, 12:03:35 - georgezgeorgez: and it's more complicated too when you have different machines with different passwords | |||
Mon, Mar 3, 2025, 12:03:38 - deeznnutz: So I was thinking we can address that in the docs | |||
Mon, Mar 3, 2025, 12:03:44 - georgezgeorgez: also if you use ssh keys but encrypt them with a password | |||
Mon, Mar 3, 2025, 12:03:53 - georgezgeorgez: so i think even if the playbooks are good and easy to use | |||
Mon, Mar 3, 2025, 12:04:03 - georgezgeorgez: we need to help some people with that connectivity bit | |||
Mon, Mar 3, 2025, 12:04:14 - georgezgeorgez: but there's also no reason, someone can't use ansible to install locally as well | |||
Mon, Mar 3, 2025, 12:04:21 - georgezgeorgez: for people just running single nodes | |||
Mon, Mar 3, 2025, 12:04:24 - georgezgeorgez: but for hyperqube | |||
Mon, Mar 3, 2025, 12:04:29 - georgezgeorgez: since they most likely have a mainnet node | |||
Mon, Mar 3, 2025, 12:04:33 - georgezgeorgez: and a hyperqube node | |||
Mon, Mar 3, 2025, 12:04:37 - georgezgeorgez: maybe an orchestrator node etc | |||
Mon, Mar 3, 2025, 12:04:40 - deeznnutz: yes. also which use to run the playbook as. I run as root so `become: true` is not needed | |||
Mon, Mar 3, 2025, 12:04:49 - deeznnutz: but I can see how that can become an issue in the future | |||
Mon, Mar 3, 2025, 12:04:51 - georgezgeorgez: being able to manage all that from a single machine/laptop is a big plus | |||
Mon, Mar 3, 2025, 12:05:13 - deeznnutz: yep. So I was thinking we do a different playbook for each "thing" | |||
Mon, Mar 3, 2025, 12:05:25 - deeznnutz: so 1 for go-zenon, 1 for go-hyperqube, etc.. | |||
Mon, Mar 3, 2025, 12:05:46 - deeznnutz: And then each playbook uses "roles" setup already so we can reuse certain "functions" over and over | |||
Mon, Mar 3, 2025, 12:05:58 - georgezgeorgez: i think that's fine for now | |||
i mentioned that the way that the matrix ansible does it, is that it uses "tags" | |||
Mon, Mar 3, 2025, 12:06:08 - georgezgeorgez: they just have one setup.yml | |||
Mon, Mar 3, 2025, 12:06:15 - georgezgeorgez: and then trigger different things with tags | |||
Mon, Mar 3, 2025, 12:06:38 - georgezgeorgez: it makes sense for them because their playbook handles installation of a bunch of plugins | |||
Mon, Mar 3, 2025, 12:06:45 - georgezgeorgez: that are all optional | |||
Mon, Mar 3, 2025, 12:06:54 - deeznnutz: I see. let me look into that | |||
Mon, Mar 3, 2025, 12:07:04 - georgezgeorgez: so instead of requiring the user to run a different playbook for each plugin | |||
they just add a tag to the command, which activates those commands | |||
Mon, Mar 3, 2025, 12:07:19 - deeznnutz: I see. that's better | |||
Mon, Mar 3, 2025, 12:07:25 - georgezgeorgez: it's good to get as much as we can into roles though | |||
so good job with that | |||
Mon, Mar 3, 2025, 12:07:29 - deeznnutz: that is what I was hoping we could get to | |||
Mon, Mar 3, 2025, 12:07:32 - georgezgeorgez: because ansible is a pretty common tool | |||
Mon, Mar 3, 2025, 12:07:43 - georgezgeorgez: and experienced operators might already have their own playbooks | |||
Mon, Mar 3, 2025, 12:07:52 - georgezgeorgez: roles can be zipped up and shared | |||
Mon, Mar 3, 2025, 12:08:06 - georgezgeorgez: so we can provide a default playbook for most people | |||
Mon, Mar 3, 2025, 12:08:09 - deeznnutz: I was looking into already standard security playbooks. | |||
Mon, Mar 3, 2025, 12:08:17 - georgezgeorgez: but anyone with their own custom playbooks can just use our roles | |||
Mon, Mar 3, 2025, 12:08:22 - deeznnutz: maybe in the future we can bring something like that in | |||
Mon, Mar 3, 2025, 12:08:41 - georgezgeorgez: do you mean like hardening? | |||
Mon, Mar 3, 2025, 12:08:46 - deeznnutz: yes | |||
Mon, Mar 3, 2025, 12:09:00 - deeznnutz: ssh hardening, and kernel hardening | |||
Mon, Mar 3, 2025, 12:09:01 - georgezgeorgez: yeah i think there are some community roles for that | |||
Mon, Mar 3, 2025, 12:09:04 - coinselor: sup fam, rocking that hotspot from mobile kek | |||
Mon, Mar 3, 2025, 12:09:35 - deeznnutz: OK so my near term focus is to get this setup to deploy and manage go-zenon and go-hyperqube. | |||
Mon, Mar 3, 2025, 12:09:44 - deeznnutz: I will be in town for a few weeks and have the time | |||
Mon, Mar 3, 2025, 12:09:55 - georgezgeorgez: not sure if you've seen | |||
Mon, Mar 3, 2025, 12:10:04 - georgezgeorgez: but ansible has operating system abstractions | |||
Mon, Mar 3, 2025, 12:10:08 - georgezgeorgez: for things like package installation | |||
Mon, Mar 3, 2025, 12:10:17 - coinselor: for passwords I saw something called vaults | |||
Mon, Mar 3, 2025, 12:10:41 - georgezgeorgez: there's an "apt" task which works with ubuntu | |||
but there's a more generic "package" task that works with any OS | |||
Mon, Mar 3, 2025, 12:10:48 - georgezgeorgez: well most OS | |||
Mon, Mar 3, 2025, 12:10:54 - deeznnutz: yes | |||
Mon, Mar 3, 2025, 12:10:56 - georgezgeorgez: as long as the package name is the same | |||
Mon, Mar 3, 2025, 12:11:01 - deeznnutz: it handles different OSs too | |||
Mon, Mar 3, 2025, 12:11:12 - deeznnutz: so it will use yum, apt, etc.. based on what you are running | |||
Mon, Mar 3, 2025, 12:11:16 - deeznnutz: it's awesome | |||
Mon, Mar 3, 2025, 12:11:21 - georgezgeorgez: so in this way, we can start supporting other OS as well | |||
Mon, Mar 3, 2025, 12:11:34 - georgezgeorgez: in terms of building go-zenon/hyperqube | |||
Mon, Mar 3, 2025, 12:11:47 - deeznnutz: The install go roll supports ARM right now. I have not tested it though | |||
Mon, Mar 3, 2025, 12:11:49 - georgezgeorgez: one thing i'm an advocate for is using the same build commands everywhere | |||
Mon, Mar 3, 2025, 12:11:56 - georgezgeorgez: we have a Makefile | |||
Mon, Mar 3, 2025, 12:12:16 - georgezgeorgez: my opinion would be to use the make commands everywhere | |||
Mon, Mar 3, 2025, 12:12:20 - georgezgeorgez: inside the github actions | |||
Mon, Mar 3, 2025, 12:12:24 - georgezgeorgez: and inside the ansible playbook | |||
Mon, Mar 3, 2025, 12:12:46 - georgezgeorgez: this way we eliminate any questions about possible differences between compilation method | |||
Mon, Mar 3, 2025, 12:13:00 - georgezgeorgez: and if we have to make a change, we just do it in the Makefile | |||
Mon, Mar 3, 2025, 12:13:16 - georgezgeorgez: <@coinselor:zenon.chat "for passwords I saw something ca..."> yes, this is something a bit more advanced | |||
Mon, Mar 3, 2025, 12:13:24 - georgezgeorgez: but definitely a feature to take advantage of | |||
Mon, Mar 3, 2025, 12:13:36 - deeznnutz: does that mean that Ansible lookes at the Makefile to "program" itself? | |||
Mon, Mar 3, 2025, 12:13:55 - georgezgeorgez: no i mean, there is an Ansible "make" task | |||
Mon, Mar 3, 2025, 12:14:18 - deeznnutz: how is that different than `ansible-playbook` | |||
Mon, Mar 3, 2025, 12:14:20 - georgezgeorgez: so that can be used | |||
instead of directly calling "go build ..." | |||
Mon, Mar 3, 2025, 12:14:26 - georgezgeorgez: sorry i mean | |||
Mon, Mar 3, 2025, 12:14:29 - coinselor: couldn't the github action technically just run an ansible playbook too? | |||
Mon, Mar 3, 2025, 12:14:32 - georgezgeorgez: inside the playbook, call `make` | |||
Mon, Mar 3, 2025, 12:15:00 - georgezgeorgez: <@coinselor:zenon.chat "couldn't the github action techn..."> the github action is on a temporary runner machine | |||
ansible is for configuring machines | |||
it's not a build system | |||
Mon, Mar 3, 2025, 12:15:10 - georgezgeorgez: but we can tell ansible to call the build system | |||
Mon, Mar 3, 2025, 12:15:21 - georgezgeorgez: and make in theory is idempotent as well | |||
Mon, Mar 3, 2025, 12:15:26 - deeznnutz: <@georgezgeorgez:hc1.chat "so that can be used"> I see | |||
Mon, Mar 3, 2025, 12:15:42 - georgezgeorgez: it's not critical, but just a suggestion | |||
that since we have the makefile | |||
Mon, Mar 3, 2025, 12:15:53 - deeznnutz: I can look into that for sure. | |||
Mon, Mar 3, 2025, 12:16:01 - georgezgeorgez: and since developers on their local machine will likely just call `make` on their machines | |||
Mon, Mar 3, 2025, 12:16:03 - deeznnutz: should be easy | |||
Mon, Mar 3, 2025, 12:16:33 - georgezgeorgez: this way we don't have to change the go build command in multiple places | |||
the makefile, the github action, the playbook | |||
Mon, Mar 3, 2025, 12:16:42 - georgezgeorgez: there might be some more considerations with cross compilation | |||
Mon, Mar 3, 2025, 12:16:51 - georgezgeorgez: but i'm guessing that can be put into a make command as well | |||
Mon, Mar 3, 2025, 12:17:08 - georgezgeorgez: and guys, this is just my opinionated way to approach it | |||
Mon, Mar 3, 2025, 12:17:19 - georgezgeorgez: if you think another way is better, then speak up as well | |||
Mon, Mar 3, 2025, 12:18:02 - deeznnutz: I'm going to look at the matrix ansible repo that guy setup. It's the most conprehensive ansible setup I've seen. I'm sure we can learn something from that | |||
Mon, Mar 3, 2025, 12:19:02 - deeznnutz: Is there anything else that is important now (or soon) to support HQZ? | |||
Mon, Mar 3, 2025, 12:19:29 - coinselor: I'm not familiar with the 'general package' task for multiple OS you mentioned, but we install very common packages: git, curl, go, etc - is ansible taking care of like finding the right package manager/packagane names for each OS? | |||
Mon, Mar 3, 2025, 12:19:38 - georgezgeorgez: <@coinselor:zenon.chat "I'm not familiar with the 'gener..."> yup | |||
Mon, Mar 3, 2025, 12:19:54 - georgezgeorgez: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/package_module.html | |||
Mon, Mar 3, 2025, 12:20:11 - deeznnutz: ya it's pretty powerful like that | |||
Mon, Mar 3, 2025, 12:20:21 - georgezgeorgez: except windows apparently | |||
Mon, Mar 3, 2025, 12:21:14 - georgezgeorgez: right now, supporting just ubuntu is probably fine | |||
Mon, Mar 3, 2025, 12:21:32 - georgezgeorgez: but long term, supporting other major OS is very important | |||
Mon, Mar 3, 2025, 12:21:44 - georgezgeorgez: if you want companies to use NoM | |||
Mon, Mar 3, 2025, 12:21:48 - deeznnutz: I can see how this will make setting up a sentry super easy | |||
Mon, Mar 3, 2025, 12:22:00 - georgezgeorgez: sentry architecture? | |||
Mon, Mar 3, 2025, 12:22:05 - deeznnutz: ya | |||
Mon, Mar 3, 2025, 12:22:19 - georgezgeorgez: yes, i would say most of our configuration tasks can be handled via ansible | |||
Mon, Mar 3, 2025, 12:22:20 - deeznnutz: like 2 sentries and 1 pillar | |||
Mon, Mar 3, 2025, 12:22:29 - georgezgeorgez: the main reason i wanted to introduce it now | |||
Mon, Mar 3, 2025, 12:22:33 - georgezgeorgez: was because the hqz community | |||
Mon, Mar 3, 2025, 12:22:38 - georgezgeorgez: are all likely running multiple nodes | |||
Mon, Mar 3, 2025, 12:22:50 - georgezgeorgez: and if we start moving really fast with hqz | |||
Mon, Mar 3, 2025, 12:23:02 - georgezgeorgez: i want to make those Ops smooth | |||
Mon, Mar 3, 2025, 12:23:10 - georgezgeorgez: in terms of what else is needed for hqz | |||
Mon, Mar 3, 2025, 12:23:21 - coinselor: we could make an interactive script that writes the required playbook variables for that right? say choosing which ip is a sentry and configuring it accordingly | |||
Mon, Mar 3, 2025, 12:23:35 - coinselor: instead of having a user edit the book | |||
Mon, Mar 3, 2025, 12:24:16 - georgezgeorgez: if we do something similar to the matrix playbook | |||
which handles everything through tags | |||
then we might want to have some stuff ready in case we have a hqz hardfork/reboot | |||
Mon, Mar 3, 2025, 12:24:18 - deeznnutz: from what I can tell you do need to identify certain things in the `inventory.yml` file. I assume IP will be one of those. that is how I'm doing it now | |||
Mon, Mar 3, 2025, 12:25:21 - georgezgeorgez: maybe we have a variable for genesis.json url | |||
and if ansible sees that it changes | |||
it backs up the dirs of the old network and creates dirs for the new ones | |||
Mon, Mar 3, 2025, 12:25:29 - georgezgeorgez: stuff like the consensus and nom folders | |||
Mon, Mar 3, 2025, 12:25:39 - georgezgeorgez: idk just thinking out loud | |||
Mon, Mar 3, 2025, 12:25:46 - georgezgeorgez: it's likely that we break hqz every now and then | |||
Mon, Mar 3, 2025, 12:25:52 - georgezgeorgez: so whatever can help us recover faster | |||
Mon, Mar 3, 2025, 12:26:04 - deeznnutz: was thinking the same thing as I plan out how to deploy the `config.json` and `genesis.json` files | |||
Mon, Mar 3, 2025, 12:26:22 - georgezgeorgez: turning the reboot process into a one liner with ansible | |||
could be the difference between rebooting in 2 weeks vs a few days | |||
Mon, Mar 3, 2025, 12:26:46 - georgezgeorgez: that's why i'm leading a bit with this operational work | |||
Mon, Mar 3, 2025, 12:26:58 - georgezgeorgez: if it was just HC1 | |||
dealing with some of those complexities as they arise | |||
Mon, Mar 3, 2025, 12:27:01 - georgezgeorgez: would probably be okay | |||
Mon, Mar 3, 2025, 12:27:07 - georgezgeorgez: but since we are dealing with wider community | |||
Mon, Mar 3, 2025, 12:27:16 - georgezgeorgez: not all of them power users | |||
Mon, Mar 3, 2025, 12:27:33 - georgezgeorgez: i want to make sure we don't lose people as we go | |||
Mon, Mar 3, 2025, 12:28:10 - coinselor: So eventually, all the user would have to do would be provision the vps, give ssh access, and start running ansible commands from their controlling vps right? | |||
Mon, Mar 3, 2025, 12:28:18 - georgezgeorgez: <@coinselor:zenon.chat "So eventually, all the user woul..."> or their laptop | |||
Mon, Mar 3, 2025, 12:28:35 - coinselor: yeah I was thinking it's not needed for the controlling node to be persistent | |||
Mon, Mar 3, 2025, 12:28:47 - deeznnutz: So I will keep working to recreate all the `deploy` script functions in ansible. Will look at tags and leveraging the`Makefile`. I'll also see what I can learn from the Matrix ansible repo. | |||
Mon, Mar 3, 2025, 12:28:48 - coinselor: I'm def just gonna run it from my main rig for now | |||
Mon, Mar 3, 2025, 12:29:40 - georgezgeorgez: i don't know if we need this | |||
but there is a testing framework for ansible | |||
https://ansible.readthedocs.io/projects/molecule/ | |||
Mon, Mar 3, 2025, 12:29:48 - deeznnutz: one thing I learned in this process, you really need a separate machine to run ansible commands, NOT from the server where the playbook does stuff. | |||
Mon, Mar 3, 2025, 12:30:03 - georgezgeorgez: it's basically a tool that you hook up with like a virtualization | |||
Mon, Mar 3, 2025, 12:30:10 - deeznnutz: I did not on accident when I first deployed matrix and my head got into a crazy loop trying to figure out WTF I was doing | |||
Mon, Mar 3, 2025, 12:30:33 - deeznnutz: <@georgezgeorgez:hc1.chat "it's basically a tool that you h..."> cool. I'll check that out | |||
Mon, Mar 3, 2025, 12:30:35 - georgezgeorgez: and then you tell it to spin up a machine and run the playbook on | |||
Mon, Mar 3, 2025, 12:30:48 - georgezgeorgez: and you can define some tests too | |||
Mon, Mar 3, 2025, 12:30:55 - georgezgeorgez: and it will run those tests | |||
and then tear the machine down | |||
Mon, Mar 3, 2025, 12:31:02 - georgezgeorgez: those machines can be docker, can be cloud | |||
Mon, Mar 3, 2025, 12:31:22 - georgezgeorgez: it's a bit advanced | |||
Mon, Mar 3, 2025, 12:31:26 - deeznnutz: O cool. It took several attempts to get these roles working | |||
Mon, Mar 3, 2025, 12:31:27 - georgezgeorgez: but just letting you know it exists | |||
Mon, Mar 3, 2025, 12:31:44 - coinselor: there's no docker image for go-zenon right? | |||
Mon, Mar 3, 2025, 12:31:53 - georgezgeorgez: idk about updated ones | |||
Mon, Mar 3, 2025, 12:31:57 - georgezgeorgez: but i have created them before | |||
Mon, Mar 3, 2025, 12:32:32 - deeznnutz: anything else we need to support HQZ? | |||
Mon, Mar 3, 2025, 12:33:15 - georgezgeorgez: maybe verify that the playbooks are backwards compatible with the script installation people have already used | |||
Mon, Mar 3, 2025, 12:33:23 - georgezgeorgez: if you run the playbook against an existing deploy | |||
Mon, Mar 3, 2025, 12:33:27 - georgezgeorgez: is it all OK with no changes? | |||
Mon, Mar 3, 2025, 12:33:40 - deeznnutz: got it. ya, was thinking about that too | |||
Mon, Mar 3, 2025, 12:33:51 - deeznnutz: I'll think about that and test | |||
Mon, Mar 3, 2025, 12:33:52 - georgezgeorgez: that way we can easily migrate over | |||
Mon, Mar 3, 2025, 12:33:59 - georgezgeorgez: once we figure out the instructions to give people for ssh | |||
Mon, Mar 3, 2025, 12:34:45 - deeznnutz: I'd like to start by assuming all users will run as root and we instruct them to use `--ask-pass` | |||
Mon, Mar 3, 2025, 12:34:52 - coinselor: yeah, good call, the scripts were a nightmare for that. Ansible should help | |||
Mon, Mar 3, 2025, 12:34:58 - deeznnutz: then move to public / private keys in the future | |||
Mon, Mar 3, 2025, 12:35:54 - georgezgeorgez: whatever you guys think makes sense | |||
Mon, Mar 3, 2025, 12:36:27 - georgezgeorgez: btw we can also get the monitoring stuff into ansible as well | |||
Mon, Mar 3, 2025, 12:37:06 - deeznnutz: Governance V1 Spec: March 9 | |||
Governance V1 Implementation & Spork Creation: March 23 | |||
Governance V1 Spork Activation: March 30 | |||
I want to make sure we are ready for this | |||
Mon, Mar 3, 2025, 12:37:27 - deeznnutz: will ansible need to do anything for these dates? | |||
Mon, Mar 3, 2025, 12:37:32 - georgezgeorgez: yeah so i think what this means, is that they need to be able to install a new version of the binary | |||
Mon, Mar 3, 2025, 12:37:37 - georgezgeorgez: between march 23 and 30 | |||
Mon, Mar 3, 2025, 12:37:56 - deeznnutz: ok so we need to be able to `upgrade` | |||
Mon, Mar 3, 2025, 12:38:17 - coinselor: Operationally, should we consider stuff like hqz votes? And I'm not specifically talking about like on-chain, I think geroge brought up nostr but I need to re read the hqz conversation. I barely skimmed through it. | |||
Maybe we could have a task for setting up a user to be ready to vote/etc | |||
Mon, Mar 3, 2025, 12:38:22 - deeznnutz: would be nice to be able to backup the chain locally too and then upgrade | |||
Mon, Mar 3, 2025, 12:38:56 - georgezgeorgez: <@coinselor:zenon.chat "Operationally, should we conside..."> haha yeah i guess separate conversation | |||
but i do think that we should start messing around with a few of us spinning up nostr servers | |||
Mon, Mar 3, 2025, 12:39:02 - georgezgeorgez: or relays rather | |||
Mon, Mar 3, 2025, 12:39:33 - georgezgeorgez: i think actually the most important use case right now would be upgrade notifications | |||
Mon, Mar 3, 2025, 12:39:49 - georgezgeorgez: it think kaine had hinted before about incorporating nostr into syrius | |||
Mon, Mar 3, 2025, 12:39:58 - georgezgeorgez: but this was for CivKit liquidity protocol iirc | |||
Mon, Mar 3, 2025, 12:40:07 - deeznnutz: should zenon.network host a server? nostr.zenon.network? | |||
Mon, Mar 3, 2025, 12:40:17 - georgezgeorgez: but i think it would be good to say | |||
configure syrius to hook up to a nostr relay | |||
Mon, Mar 3, 2025, 12:40:24 - georgezgeorgez: which only sends out major updates | |||
Mon, Mar 3, 2025, 12:40:31 - coinselor: I can look into automating upgrade notifications, maybe looking at spork creation on-chain? | |||
Mon, Mar 3, 2025, 12:40:33 - georgezgeorgez: like "There's an upgrade available for Syrius" | |||
Mon, Mar 3, 2025, 12:40:51 - georgezgeorgez: sporks are really just for protocol upgrades | |||
Mon, Mar 3, 2025, 12:41:07 - georgezgeorgez: not necessarily great as a comm protocol | |||
Mon, Mar 3, 2025, 12:41:28 - deeznnutz: wonder if we could pull messages into syrius? | |||
Mon, Mar 3, 2025, 12:41:39 - georgezgeorgez: yes why not? if syrius had a nostr client into it | |||
Mon, Mar 3, 2025, 12:41:49 - georgezgeorgez: i'm thinking of how to get critical messages to users | |||
Mon, Mar 3, 2025, 12:42:05 - deeznnutz: that could actually be a good way to communicate with everyone | |||
Mon, Mar 3, 2025, 12:42:05 - georgezgeorgez: maybe most users won't check Telegram, matrix, or forums | |||
Mon, Mar 3, 2025, 12:42:14 - georgezgeorgez: but if they are opening wallet to check rewards | |||
Mon, Mar 3, 2025, 12:42:28 - georgezgeorgez: if they only see 1 important message there | |||
Mon, Mar 3, 2025, 12:42:54 - georgezgeorgez: definitely only notifications that likely require action | |||
Mon, Mar 3, 2025, 12:42:56 - coinselor: I think this should be automated: | |||
- On github release -> post on all channels | |||
- On on-chain spork -> post on all channels | |||
etc | |||
Mon, Mar 3, 2025, 12:43:08 - georgezgeorgez: yes that's a good idea as well | |||
Mon, Mar 3, 2025, 12:43:15 - georgezgeorgez: could have some bots that monitor github and on-chain | |||
Mon, Mar 3, 2025, 12:43:17 - deeznnutz: ya with a GhA | |||
Mon, Mar 3, 2025, 12:43:56 - georgezgeorgez: but let's say just for HQZ for now | |||
Mon, Mar 3, 2025, 12:44:11 - georgezgeorgez: not all hqz pillars have a matrix or even a forum user iirc | |||
Mon, Mar 3, 2025, 12:44:20 - coinselor: and let users choose what the notifications they like best: x bot, tg bot, discord bot, matrix bot, etc | |||
Mon, Mar 3, 2025, 12:44:23 - georgezgeorgez: we are reaching them through AZ votes | |||
Mon, Mar 3, 2025, 12:44:25 - coinselor: nostr bot kek | |||
Mon, Mar 3, 2025, 12:44:29 - georgezgeorgez: but we can't do that forever | |||
Mon, Mar 3, 2025, 12:45:08 - georgezgeorgez: i'm thinking the minimum engagement is reward collecting | |||
Mon, Mar 3, 2025, 12:45:22 - georgezgeorgez: so if we can meet them there, we'll cast the widest net | |||
Mon, Mar 3, 2025, 12:45:24 - coinselor: show an ad on collect tab | |||
Mon, Mar 3, 2025, 12:45:25 - deeznnutz: is this something we should bring up in the Syrius SIG? maybe John M. would be interested in this | |||
Mon, Mar 3, 2025, 12:45:35 - deeznnutz: or CF | |||
Mon, Mar 3, 2025, 12:46:16 - georgezgeorgez: sure why not | |||
i think there will be some questions like | |||
should syrius be a generic nostr client? | |||
or should we have sort of protocol on top to indicate message priority | |||
Mon, Mar 3, 2025, 12:46:26 - coinselor: yeah I think syrius should prompt users whenever a new version is available | |||
Mon, Mar 3, 2025, 12:46:44 - georgezgeorgez: the question is how does syrius get that data | |||
Mon, Mar 3, 2025, 12:46:51 - georgezgeorgez: to know there's a new version | |||
Mon, Mar 3, 2025, 12:46:53 - coinselor: yeah I assume it's not trivial | |||
Mon, Mar 3, 2025, 12:47:00 - georgezgeorgez: centralized server? | |||
Mon, Mar 3, 2025, 12:47:07 - deeznnutz: wonder if that becomes an attack vector? | |||
Mon, Mar 3, 2025, 12:47:12 - coinselor: I've seen some crazy stuff for like new git versions with multiple devs signing stuff | |||
Mon, Mar 3, 2025, 12:47:23 - georgezgeorgez: well it's similar to our default rpc node selection right now | |||
Mon, Mar 3, 2025, 12:47:34 - georgezgeorgez: i think HC1 could run a nostr server | |||
Mon, Mar 3, 2025, 12:47:39 - georgezgeorgez: and we could set that as a default in syrius | |||
Mon, Mar 3, 2025, 12:47:51 - georgezgeorgez: but the option is there to change it | |||
Mon, Mar 3, 2025, 12:48:04 - georgezgeorgez: i mean with nostr, in theory you can subscribe to multiple relays | |||
Mon, Mar 3, 2025, 12:48:09 - coinselor: Technically we could have a frosted nostr multisig even to post messages | |||
Mon, Mar 3, 2025, 12:48:15 - georgezgeorgez: so i've been thinking about this and prototyping in nomctl | |||
Mon, Mar 3, 2025, 12:48:23 - georgezgeorgez: since for a while, that will be the hqz wallet | |||
Mon, Mar 3, 2025, 12:48:24 - deeznnutz: yep and we can hold a `.json` file that has nostr relays that people can submit PRs to add / remove them | |||
Mon, Mar 3, 2025, 12:48:28 - georgezgeorgez: and we need a fast way to tell hqz pillars | |||
Mon, Mar 3, 2025, 12:48:34 - georgezgeorgez: hey upgrade within a week please | |||
Mon, Mar 3, 2025, 12:48:54 - deeznnutz: cant remember, can we change the chainID in syrius now? | |||
Mon, Mar 3, 2025, 12:49:01 - deeznnutz: can we setup syrius to use HQZ? | |||
Mon, Mar 3, 2025, 12:49:14 - coinselor: I think so | |||
Mon, Mar 3, 2025, 12:49:29 - georgezgeorgez: i don't think we'll have most of the new functionality on syrius | |||
Mon, Mar 3, 2025, 12:49:35 - georgezgeorgez: until it's finalized on hqz | |||
Mon, Mar 3, 2025, 12:49:38 - georgezgeorgez: and ready for mainnet | |||
Mon, Mar 3, 2025, 12:49:53 - georgezgeorgez: i don't think we want to iterate over and over again with syrius | |||
Mon, Mar 3, 2025, 12:50:14 - georgezgeorgez: this is another reason i've been wanting syrius to have a "plugin" architecture | |||
Mon, Mar 3, 2025, 12:50:16 - georgezgeorgez: if possible | |||
Mon, Mar 3, 2025, 12:50:28 - georgezgeorgez: but it seems like from a Flutter perspective, it's not straightforward | |||
Mon, Mar 3, 2025, 12:50:43 - coinselor: yeah that makes no sense, but for stuff we haven't built like a notification system, we could iterate with hqz notifications and tweak the system until we are happy and then we just change node/chain id and it should work for nom too | |||
Mon, Mar 3, 2025, 12:51:01 - georgezgeorgez: like imagine you connect your wallet to a chain | |||
it detects all the supported features, and then configures the UI with the widgets that work for that chain | |||
Mon, Mar 3, 2025, 12:51:21 - deeznnutz: that would be cool | |||
Mon, Mar 3, 2025, 12:51:23 - georgezgeorgez: but long term dreams lol | |||
Mon, Mar 3, 2025, 12:51:54 - georgezgeorgez: for now, i'm just going to build out UI on nomctl | |||
and if the contracts are good on hqz | |||
then we can port it over to syrius | |||
Mon, Mar 3, 2025, 12:52:05 - georgezgeorgez: it's a lot easier to iterate UI on CLI/TUI | |||
Mon, Mar 3, 2025, 12:52:13 - coinselor: that seems doable, just check if the chain has liquidity embedded to show staking tab | |||
Mon, Mar 3, 2025, 12:53:05 - deeznnutz: I'm bumping up on my next meeting. I know what to do and will keep everyone informed as I make progress | |||
Mon, Mar 3, 2025, 12:53:06 - coinselor: nvm embeddeds have diff addresses on diff chains? | |||
Mon, Mar 3, 2025, 12:53:33 - georgezgeorgez: not necessarily, the addresses are the same on mainnet and hqz | |||
Mon, Mar 3, 2025, 12:54:22 - georgezgeorgez: lol, it's really just imagination talk right now | |||
Mon, Mar 3, 2025, 12:55:06 - georgezgeorgez: but for example each chain could have metadata like | |||
here's a list of contracts: | |||
their addresses | |||
and some sort of UI specification for it | |||
Mon, Mar 3, 2025, 12:55:20 - georgezgeorgez: then the wallet can match up certain widgets to those different UI specifications | |||
Mon, Mar 3, 2025, 12:55:25 - coinselor: well I'm sure sooner rather than later we'll need an ez to setup local nom testnet. I know you've done it but it's not trivial for new devz i think | |||
Mon, Mar 3, 2025, 12:55:47 - georgezgeorgez: using ansible that should be fairly simple actually | |||
Mon, Mar 3, 2025, 12:56:23 - deeznnutz: i have one last question | |||
Mon, Mar 3, 2025, 12:56:34 - deeznnutz: what is the best way to make a wallet (producer) with ansible? | |||
Mon, Mar 3, 2025, 12:56:34 - georgezgeorgez: yes and we are hitting the end of the hour | |||
Mon, Mar 3, 2025, 12:56:48 - deeznnutz: dart cli tool and make a wallet called `producer` | |||
Mon, Mar 3, 2025, 12:56:49 - deeznnutz: ? | |||
Mon, Mar 3, 2025, 12:57:06 - georgezgeorgez: <@deeznnutz:zenon.chat "what is the best way to make a w..."> probably just call a CLI like dart tool or nomctl | |||
Mon, Mar 3, 2025, 12:57:30 - georgezgeorgez: i don't think we want ansible to worry about cryptographic logic | |||
Mon, Mar 3, 2025, 12:57:32 - deeznnutz: ya, so download the cli from GH and run the command to make the wallet | |||
Mon, Mar 3, 2025, 12:57:38 - georgezgeorgez: just install the cli | |||
then run the command | |||
Mon, Mar 3, 2025, 12:57:41 - georgezgeorgez: yah | |||
Mon, Mar 3, 2025, 12:57:48 - georgezgeorgez: and then after that, i think everything else is just templated files | |||
Mon, Mar 3, 2025, 12:57:50 - deeznnutz: ok - super easy | |||
Mon, Mar 3, 2025, 12:57:59 - georgezgeorgez: i think for nodes, there's also the network key | |||
Mon, Mar 3, 2025, 12:58:02 - coinselor: I should be back in full form in a few days, I'll help if you can create/list some pending tasks ^^ | |||
Mon, Mar 3, 2025, 12:58:23 - georgezgeorgez: <@georgezgeorgez:hc1.chat "i think for nodes, there's also ..."> maybe for nomctl i can create a utility command which just creates that | |||
Mon, Mar 3, 2025, 12:58:36 - georgezgeorgez: this way we can setup the entire file structure without having to start up the node first | |||
Mon, Mar 3, 2025, 12:58:48 - deeznnutz: ya that would be helpful for sure | |||
Mon, Mar 3, 2025, 12:59:03 - deeznnutz: what is the network key? | |||
Mon, Mar 3, 2025, 12:59:07 - deeznnutz: not familiar with that | |||
Mon, Mar 3, 2025, 12:59:35 - coinselor: i think he means the node id ? | |||
Mon, Mar 3, 2025, 12:59:37 - georgezgeorgez: it's the key that is used for the enode | |||
i think in the folder it's literally a file called like network-key | |||
Mon, Mar 3, 2025, 12:59:56 - georgezgeorgez: so there's a producer key, which signs created transactions | |||
Mon, Mar 3, 2025, 13:00:03 - deeznnutz: Something that Pillars need? | |||
Mon, Mar 3, 2025, 13:00:12 - georgezgeorgez: and then each node has a key used for the p2p network | |||
which yeah, gets represented in node id | |||
Mon, Mar 3, 2025, 13:00:24 - deeznnutz: I did not know that | |||
Mon, Mar 3, 2025, 13:00:27 - georgezgeorgez: all nodes create one when they start up if it isn't there i think | |||
Mon, Mar 3, 2025, 13:00:30 - deeznnutz: I'll look into it | |||
Mon, Mar 3, 2025, 13:00:43 - georgezgeorgez: it's used for example to verify you haven't been man in the middled | |||
Mon, Mar 3, 2025, 13:00:45 - coinselor: yeah supernova has something similar i think | |||
Mon, Mar 3, 2025, 13:00:53 - georgezgeorgez: and that the node you're talking to on this IP | |||
Mon, Mar 3, 2025, 13:00:56 - georgezgeorgez: is actually that one | |||
Mon, Mar 3, 2025, 13:01:06 - deeznnutz: I see | |||
Mon, Mar 3, 2025, 13:01:17 - deeznnutz: prevents spoofing the IP | |||
Mon, Mar 3, 2025, 13:01:37 - georgezgeorgez: yeah stuff like that | |||
would have to go deep into the p2p stack to cover all its functionality | |||
Mon, Mar 3, 2025, 13:01:49 - georgezgeorgez: and once we switch to libp2p | |||
that file might go away/change | |||
Mon, Mar 3, 2025, 13:02:06 - georgezgeorgez: but first major upgrade | |||
functionality focused | |||
Mon, Mar 3, 2025, 13:02:19 - deeznnutz: cool. I'm all set. | |||
Mon, Mar 3, 2025, 13:02:32 - georgezgeorgez: awesome. thanks everyone | |||
good meeting | |||
Mon, Mar 3, 2025, 13:02:47 - coinselor: thank you all | |||
Mon, Mar 3, 2025, 13:02:49 - deeznnutz: thanks everyone.!!!! great meeting | |||
Mon, Mar 3, 2025, 13:02:57 - deeznnutz: ==== OP SIG END ==== |
Revision as of 22:31, 4 March 2025
Agenda
What: Meeting to Discuss Improving Node Operations as part of the HC1: OP SIG
When: 03 March 2025 @ 8 CET EST
Where: https://matrix.to/#/#sig-op:hc1.chat
Chair: 0x3639
Agenda:
Discuss follow Up items from previous meeting Document action items Establish next meeting
If you want to attend please respond (or DM) with your full matrix username and I will invite you to the group. No FUD, anger or BS allowed.
Pre-meeting Notes
- Introduced Ansible to 0x and how we can use it for multi platform deployments
Meeting Minutes Summary (chatGPT)
Meeting Minutes
Mon, Mar 3, 2025, 11:59:57 - deeznnutz: ==== OG SIG Start ==== Mon, Mar 3, 2025, 12:00:17 - deeznnutz: GM. I dont have anything prepared to start with, but did make progress on ansible Mon, Mar 3, 2025, 12:00:36 - deeznnutz: It easy to see how this will be very helpful and way more powerful than the scripts we were using before Mon, Mar 3, 2025, 12:00:55 - deeznnutz: georgezgeorgez: did you take and look and any initial feedback? Was I using Roles correctly? Mon, Mar 3, 2025, 12:01:17 - georgezgeorgez: i took an initial look but haven't gone through everything yet Mon, Mar 3, 2025, 12:01:24 - georgezgeorgez: i think what's there is good Mon, Mar 3, 2025, 12:01:33 - georgezgeorgez: i might make some PRs in the coming weeks Mon, Mar 3, 2025, 12:01:41 - georgezgeorgez: but otherwise keep it up Mon, Mar 3, 2025, 12:01:50 - georgezgeorgez: there are some concepts like "handlers" Mon, Mar 3, 2025, 12:01:59 - georgezgeorgez: that we can take advantage as well Mon, Mar 3, 2025, 12:02:11 - georgezgeorgez: when refamiliarizing myself with ansible last week Mon, Mar 3, 2025, 12:02:23 - georgezgeorgez: i think one tricky part is actually not part of these playbooks Mon, Mar 3, 2025, 12:02:30 - georgezgeorgez: it's the setting up of the ssh connection Mon, Mar 3, 2025, 12:02:36 - deeznnutz: lol Mon, Mar 3, 2025, 12:02:38 - deeznnutz: yes!! Mon, Mar 3, 2025, 12:02:47 - georgezgeorgez: Ansible prefers ssh keypairs Mon, Mar 3, 2025, 12:03:10 - georgezgeorgez: but i'm guessing a lot of people are just using passwords Mon, Mar 3, 2025, 12:03:18 - deeznnutz: yes I was going to mention that. I used `--ask-pass` to get around that but it requires local packages to use that flag Mon, Mar 3, 2025, 12:03:26 - georgezgeorgez: yes exactly Mon, Mar 3, 2025, 12:03:35 - georgezgeorgez: and it's more complicated too when you have different machines with different passwords Mon, Mar 3, 2025, 12:03:38 - deeznnutz: So I was thinking we can address that in the docs Mon, Mar 3, 2025, 12:03:44 - georgezgeorgez: also if you use ssh keys but encrypt them with a password Mon, Mar 3, 2025, 12:03:53 - georgezgeorgez: so i think even if the playbooks are good and easy to use Mon, Mar 3, 2025, 12:04:03 - georgezgeorgez: we need to help some people with that connectivity bit Mon, Mar 3, 2025, 12:04:14 - georgezgeorgez: but there's also no reason, someone can't use ansible to install locally as well Mon, Mar 3, 2025, 12:04:21 - georgezgeorgez: for people just running single nodes Mon, Mar 3, 2025, 12:04:24 - georgezgeorgez: but for hyperqube Mon, Mar 3, 2025, 12:04:29 - georgezgeorgez: since they most likely have a mainnet node Mon, Mar 3, 2025, 12:04:33 - georgezgeorgez: and a hyperqube node Mon, Mar 3, 2025, 12:04:37 - georgezgeorgez: maybe an orchestrator node etc Mon, Mar 3, 2025, 12:04:40 - deeznnutz: yes. also which use to run the playbook as. I run as root so `become: true` is not needed Mon, Mar 3, 2025, 12:04:49 - deeznnutz: but I can see how that can become an issue in the future Mon, Mar 3, 2025, 12:04:51 - georgezgeorgez: being able to manage all that from a single machine/laptop is a big plus Mon, Mar 3, 2025, 12:05:13 - deeznnutz: yep. So I was thinking we do a different playbook for each "thing" Mon, Mar 3, 2025, 12:05:25 - deeznnutz: so 1 for go-zenon, 1 for go-hyperqube, etc.. Mon, Mar 3, 2025, 12:05:46 - deeznnutz: And then each playbook uses "roles" setup already so we can reuse certain "functions" over and over Mon, Mar 3, 2025, 12:05:58 - georgezgeorgez: i think that's fine for now i mentioned that the way that the matrix ansible does it, is that it uses "tags" Mon, Mar 3, 2025, 12:06:08 - georgezgeorgez: they just have one setup.yml Mon, Mar 3, 2025, 12:06:15 - georgezgeorgez: and then trigger different things with tags Mon, Mar 3, 2025, 12:06:38 - georgezgeorgez: it makes sense for them because their playbook handles installation of a bunch of plugins Mon, Mar 3, 2025, 12:06:45 - georgezgeorgez: that are all optional Mon, Mar 3, 2025, 12:06:54 - deeznnutz: I see. let me look into that Mon, Mar 3, 2025, 12:07:04 - georgezgeorgez: so instead of requiring the user to run a different playbook for each plugin they just add a tag to the command, which activates those commands Mon, Mar 3, 2025, 12:07:19 - deeznnutz: I see. that's better Mon, Mar 3, 2025, 12:07:25 - georgezgeorgez: it's good to get as much as we can into roles though so good job with that Mon, Mar 3, 2025, 12:07:29 - deeznnutz: that is what I was hoping we could get to Mon, Mar 3, 2025, 12:07:32 - georgezgeorgez: because ansible is a pretty common tool Mon, Mar 3, 2025, 12:07:43 - georgezgeorgez: and experienced operators might already have their own playbooks Mon, Mar 3, 2025, 12:07:52 - georgezgeorgez: roles can be zipped up and shared Mon, Mar 3, 2025, 12:08:06 - georgezgeorgez: so we can provide a default playbook for most people Mon, Mar 3, 2025, 12:08:09 - deeznnutz: I was looking into already standard security playbooks. Mon, Mar 3, 2025, 12:08:17 - georgezgeorgez: but anyone with their own custom playbooks can just use our roles Mon, Mar 3, 2025, 12:08:22 - deeznnutz: maybe in the future we can bring something like that in Mon, Mar 3, 2025, 12:08:41 - georgezgeorgez: do you mean like hardening? Mon, Mar 3, 2025, 12:08:46 - deeznnutz: yes Mon, Mar 3, 2025, 12:09:00 - deeznnutz: ssh hardening, and kernel hardening Mon, Mar 3, 2025, 12:09:01 - georgezgeorgez: yeah i think there are some community roles for that Mon, Mar 3, 2025, 12:09:04 - coinselor: sup fam, rocking that hotspot from mobile kek Mon, Mar 3, 2025, 12:09:35 - deeznnutz: OK so my near term focus is to get this setup to deploy and manage go-zenon and go-hyperqube. Mon, Mar 3, 2025, 12:09:44 - deeznnutz: I will be in town for a few weeks and have the time Mon, Mar 3, 2025, 12:09:55 - georgezgeorgez: not sure if you've seen Mon, Mar 3, 2025, 12:10:04 - georgezgeorgez: but ansible has operating system abstractions Mon, Mar 3, 2025, 12:10:08 - georgezgeorgez: for things like package installation Mon, Mar 3, 2025, 12:10:17 - coinselor: for passwords I saw something called vaults Mon, Mar 3, 2025, 12:10:41 - georgezgeorgez: there's an "apt" task which works with ubuntu but there's a more generic "package" task that works with any OS Mon, Mar 3, 2025, 12:10:48 - georgezgeorgez: well most OS Mon, Mar 3, 2025, 12:10:54 - deeznnutz: yes Mon, Mar 3, 2025, 12:10:56 - georgezgeorgez: as long as the package name is the same Mon, Mar 3, 2025, 12:11:01 - deeznnutz: it handles different OSs too Mon, Mar 3, 2025, 12:11:12 - deeznnutz: so it will use yum, apt, etc.. based on what you are running Mon, Mar 3, 2025, 12:11:16 - deeznnutz: it's awesome Mon, Mar 3, 2025, 12:11:21 - georgezgeorgez: so in this way, we can start supporting other OS as well Mon, Mar 3, 2025, 12:11:34 - georgezgeorgez: in terms of building go-zenon/hyperqube Mon, Mar 3, 2025, 12:11:47 - deeznnutz: The install go roll supports ARM right now. I have not tested it though Mon, Mar 3, 2025, 12:11:49 - georgezgeorgez: one thing i'm an advocate for is using the same build commands everywhere Mon, Mar 3, 2025, 12:11:56 - georgezgeorgez: we have a Makefile Mon, Mar 3, 2025, 12:12:16 - georgezgeorgez: my opinion would be to use the make commands everywhere Mon, Mar 3, 2025, 12:12:20 - georgezgeorgez: inside the github actions Mon, Mar 3, 2025, 12:12:24 - georgezgeorgez: and inside the ansible playbook Mon, Mar 3, 2025, 12:12:46 - georgezgeorgez: this way we eliminate any questions about possible differences between compilation method Mon, Mar 3, 2025, 12:13:00 - georgezgeorgez: and if we have to make a change, we just do it in the Makefile Mon, Mar 3, 2025, 12:13:16 - georgezgeorgez: <@coinselor:zenon.chat "for passwords I saw something ca..."> yes, this is something a bit more advanced Mon, Mar 3, 2025, 12:13:24 - georgezgeorgez: but definitely a feature to take advantage of Mon, Mar 3, 2025, 12:13:36 - deeznnutz: does that mean that Ansible lookes at the Makefile to "program" itself? Mon, Mar 3, 2025, 12:13:55 - georgezgeorgez: no i mean, there is an Ansible "make" task Mon, Mar 3, 2025, 12:14:18 - deeznnutz: how is that different than `ansible-playbook` Mon, Mar 3, 2025, 12:14:20 - georgezgeorgez: so that can be used instead of directly calling "go build ..." Mon, Mar 3, 2025, 12:14:26 - georgezgeorgez: sorry i mean Mon, Mar 3, 2025, 12:14:29 - coinselor: couldn't the github action technically just run an ansible playbook too? Mon, Mar 3, 2025, 12:14:32 - georgezgeorgez: inside the playbook, call `make` Mon, Mar 3, 2025, 12:15:00 - georgezgeorgez: <@coinselor:zenon.chat "couldn't the github action techn..."> the github action is on a temporary runner machine ansible is for configuring machines it's not a build system Mon, Mar 3, 2025, 12:15:10 - georgezgeorgez: but we can tell ansible to call the build system Mon, Mar 3, 2025, 12:15:21 - georgezgeorgez: and make in theory is idempotent as well Mon, Mar 3, 2025, 12:15:26 - deeznnutz: <@georgezgeorgez:hc1.chat "so that can be used"> I see Mon, Mar 3, 2025, 12:15:42 - georgezgeorgez: it's not critical, but just a suggestion that since we have the makefile Mon, Mar 3, 2025, 12:15:53 - deeznnutz: I can look into that for sure. Mon, Mar 3, 2025, 12:16:01 - georgezgeorgez: and since developers on their local machine will likely just call `make` on their machines Mon, Mar 3, 2025, 12:16:03 - deeznnutz: should be easy Mon, Mar 3, 2025, 12:16:33 - georgezgeorgez: this way we don't have to change the go build command in multiple places the makefile, the github action, the playbook Mon, Mar 3, 2025, 12:16:42 - georgezgeorgez: there might be some more considerations with cross compilation Mon, Mar 3, 2025, 12:16:51 - georgezgeorgez: but i'm guessing that can be put into a make command as well Mon, Mar 3, 2025, 12:17:08 - georgezgeorgez: and guys, this is just my opinionated way to approach it Mon, Mar 3, 2025, 12:17:19 - georgezgeorgez: if you think another way is better, then speak up as well Mon, Mar 3, 2025, 12:18:02 - deeznnutz: I'm going to look at the matrix ansible repo that guy setup. It's the most conprehensive ansible setup I've seen. I'm sure we can learn something from that Mon, Mar 3, 2025, 12:19:02 - deeznnutz: Is there anything else that is important now (or soon) to support HQZ? Mon, Mar 3, 2025, 12:19:29 - coinselor: I'm not familiar with the 'general package' task for multiple OS you mentioned, but we install very common packages: git, curl, go, etc - is ansible taking care of like finding the right package manager/packagane names for each OS?
Mon, Mar 3, 2025, 12:19:38 - georgezgeorgez: <@coinselor:zenon.chat "I'm not familiar with the 'gener..."> yup Mon, Mar 3, 2025, 12:19:54 - georgezgeorgez: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/package_module.html Mon, Mar 3, 2025, 12:20:11 - deeznnutz: ya it's pretty powerful like that Mon, Mar 3, 2025, 12:20:21 - georgezgeorgez: except windows apparently Mon, Mar 3, 2025, 12:21:14 - georgezgeorgez: right now, supporting just ubuntu is probably fine Mon, Mar 3, 2025, 12:21:32 - georgezgeorgez: but long term, supporting other major OS is very important Mon, Mar 3, 2025, 12:21:44 - georgezgeorgez: if you want companies to use NoM Mon, Mar 3, 2025, 12:21:48 - deeznnutz: I can see how this will make setting up a sentry super easy Mon, Mar 3, 2025, 12:22:00 - georgezgeorgez: sentry architecture? Mon, Mar 3, 2025, 12:22:05 - deeznnutz: ya Mon, Mar 3, 2025, 12:22:19 - georgezgeorgez: yes, i would say most of our configuration tasks can be handled via ansible Mon, Mar 3, 2025, 12:22:20 - deeznnutz: like 2 sentries and 1 pillar Mon, Mar 3, 2025, 12:22:29 - georgezgeorgez: the main reason i wanted to introduce it now Mon, Mar 3, 2025, 12:22:33 - georgezgeorgez: was because the hqz community Mon, Mar 3, 2025, 12:22:38 - georgezgeorgez: are all likely running multiple nodes Mon, Mar 3, 2025, 12:22:50 - georgezgeorgez: and if we start moving really fast with hqz Mon, Mar 3, 2025, 12:23:02 - georgezgeorgez: i want to make those Ops smooth Mon, Mar 3, 2025, 12:23:10 - georgezgeorgez: in terms of what else is needed for hqz Mon, Mar 3, 2025, 12:23:21 - coinselor: we could make an interactive script that writes the required playbook variables for that right? say choosing which ip is a sentry and configuring it accordingly Mon, Mar 3, 2025, 12:23:35 - coinselor: instead of having a user edit the book Mon, Mar 3, 2025, 12:24:16 - georgezgeorgez: if we do something similar to the matrix playbook which handles everything through tags then we might want to have some stuff ready in case we have a hqz hardfork/reboot Mon, Mar 3, 2025, 12:24:18 - deeznnutz: from what I can tell you do need to identify certain things in the `inventory.yml` file. I assume IP will be one of those. that is how I'm doing it now Mon, Mar 3, 2025, 12:25:21 - georgezgeorgez: maybe we have a variable for genesis.json url and if ansible sees that it changes it backs up the dirs of the old network and creates dirs for the new ones Mon, Mar 3, 2025, 12:25:29 - georgezgeorgez: stuff like the consensus and nom folders Mon, Mar 3, 2025, 12:25:39 - georgezgeorgez: idk just thinking out loud Mon, Mar 3, 2025, 12:25:46 - georgezgeorgez: it's likely that we break hqz every now and then Mon, Mar 3, 2025, 12:25:52 - georgezgeorgez: so whatever can help us recover faster Mon, Mar 3, 2025, 12:26:04 - deeznnutz: was thinking the same thing as I plan out how to deploy the `config.json` and `genesis.json` files Mon, Mar 3, 2025, 12:26:22 - georgezgeorgez: turning the reboot process into a one liner with ansible could be the difference between rebooting in 2 weeks vs a few days Mon, Mar 3, 2025, 12:26:46 - georgezgeorgez: that's why i'm leading a bit with this operational work Mon, Mar 3, 2025, 12:26:58 - georgezgeorgez: if it was just HC1 dealing with some of those complexities as they arise Mon, Mar 3, 2025, 12:27:01 - georgezgeorgez: would probably be okay Mon, Mar 3, 2025, 12:27:07 - georgezgeorgez: but since we are dealing with wider community Mon, Mar 3, 2025, 12:27:16 - georgezgeorgez: not all of them power users Mon, Mar 3, 2025, 12:27:33 - georgezgeorgez: i want to make sure we don't lose people as we go Mon, Mar 3, 2025, 12:28:10 - coinselor: So eventually, all the user would have to do would be provision the vps, give ssh access, and start running ansible commands from their controlling vps right?
Mon, Mar 3, 2025, 12:28:18 - georgezgeorgez: <@coinselor:zenon.chat "So eventually, all the user woul..."> or their laptop Mon, Mar 3, 2025, 12:28:35 - coinselor: yeah I was thinking it's not needed for the controlling node to be persistent Mon, Mar 3, 2025, 12:28:47 - deeznnutz: So I will keep working to recreate all the `deploy` script functions in ansible. Will look at tags and leveraging the`Makefile`. I'll also see what I can learn from the Matrix ansible repo. Mon, Mar 3, 2025, 12:28:48 - coinselor: I'm def just gonna run it from my main rig for now Mon, Mar 3, 2025, 12:29:40 - georgezgeorgez: i don't know if we need this but there is a testing framework for ansible https://ansible.readthedocs.io/projects/molecule/ Mon, Mar 3, 2025, 12:29:48 - deeznnutz: one thing I learned in this process, you really need a separate machine to run ansible commands, NOT from the server where the playbook does stuff. Mon, Mar 3, 2025, 12:30:03 - georgezgeorgez: it's basically a tool that you hook up with like a virtualization Mon, Mar 3, 2025, 12:30:10 - deeznnutz: I did not on accident when I first deployed matrix and my head got into a crazy loop trying to figure out WTF I was doing Mon, Mar 3, 2025, 12:30:33 - deeznnutz: <@georgezgeorgez:hc1.chat "it's basically a tool that you h..."> cool. I'll check that out Mon, Mar 3, 2025, 12:30:35 - georgezgeorgez: and then you tell it to spin up a machine and run the playbook on Mon, Mar 3, 2025, 12:30:48 - georgezgeorgez: and you can define some tests too Mon, Mar 3, 2025, 12:30:55 - georgezgeorgez: and it will run those tests and then tear the machine down Mon, Mar 3, 2025, 12:31:02 - georgezgeorgez: those machines can be docker, can be cloud Mon, Mar 3, 2025, 12:31:22 - georgezgeorgez: it's a bit advanced Mon, Mar 3, 2025, 12:31:26 - deeznnutz: O cool. It took several attempts to get these roles working Mon, Mar 3, 2025, 12:31:27 - georgezgeorgez: but just letting you know it exists Mon, Mar 3, 2025, 12:31:44 - coinselor: there's no docker image for go-zenon right? Mon, Mar 3, 2025, 12:31:53 - georgezgeorgez: idk about updated ones Mon, Mar 3, 2025, 12:31:57 - georgezgeorgez: but i have created them before Mon, Mar 3, 2025, 12:32:32 - deeznnutz: anything else we need to support HQZ? Mon, Mar 3, 2025, 12:33:15 - georgezgeorgez: maybe verify that the playbooks are backwards compatible with the script installation people have already used Mon, Mar 3, 2025, 12:33:23 - georgezgeorgez: if you run the playbook against an existing deploy Mon, Mar 3, 2025, 12:33:27 - georgezgeorgez: is it all OK with no changes? Mon, Mar 3, 2025, 12:33:40 - deeznnutz: got it. ya, was thinking about that too Mon, Mar 3, 2025, 12:33:51 - deeznnutz: I'll think about that and test Mon, Mar 3, 2025, 12:33:52 - georgezgeorgez: that way we can easily migrate over Mon, Mar 3, 2025, 12:33:59 - georgezgeorgez: once we figure out the instructions to give people for ssh Mon, Mar 3, 2025, 12:34:45 - deeznnutz: I'd like to start by assuming all users will run as root and we instruct them to use `--ask-pass` Mon, Mar 3, 2025, 12:34:52 - coinselor: yeah, good call, the scripts were a nightmare for that. Ansible should help Mon, Mar 3, 2025, 12:34:58 - deeznnutz: then move to public / private keys in the future Mon, Mar 3, 2025, 12:35:54 - georgezgeorgez: whatever you guys think makes sense Mon, Mar 3, 2025, 12:36:27 - georgezgeorgez: btw we can also get the monitoring stuff into ansible as well Mon, Mar 3, 2025, 12:37:06 - deeznnutz: Governance V1 Spec: March 9 Governance V1 Implementation & Spork Creation: March 23 Governance V1 Spork Activation: March 30
I want to make sure we are ready for this Mon, Mar 3, 2025, 12:37:27 - deeznnutz: will ansible need to do anything for these dates? Mon, Mar 3, 2025, 12:37:32 - georgezgeorgez: yeah so i think what this means, is that they need to be able to install a new version of the binary Mon, Mar 3, 2025, 12:37:37 - georgezgeorgez: between march 23 and 30 Mon, Mar 3, 2025, 12:37:56 - deeznnutz: ok so we need to be able to `upgrade` Mon, Mar 3, 2025, 12:38:17 - coinselor: Operationally, should we consider stuff like hqz votes? And I'm not specifically talking about like on-chain, I think geroge brought up nostr but I need to re read the hqz conversation. I barely skimmed through it.
Maybe we could have a task for setting up a user to be ready to vote/etc Mon, Mar 3, 2025, 12:38:22 - deeznnutz: would be nice to be able to backup the chain locally too and then upgrade Mon, Mar 3, 2025, 12:38:56 - georgezgeorgez: <@coinselor:zenon.chat "Operationally, should we conside..."> haha yeah i guess separate conversation but i do think that we should start messing around with a few of us spinning up nostr servers Mon, Mar 3, 2025, 12:39:02 - georgezgeorgez: or relays rather Mon, Mar 3, 2025, 12:39:33 - georgezgeorgez: i think actually the most important use case right now would be upgrade notifications Mon, Mar 3, 2025, 12:39:49 - georgezgeorgez: it think kaine had hinted before about incorporating nostr into syrius Mon, Mar 3, 2025, 12:39:58 - georgezgeorgez: but this was for CivKit liquidity protocol iirc Mon, Mar 3, 2025, 12:40:07 - deeznnutz: should zenon.network host a server? nostr.zenon.network? Mon, Mar 3, 2025, 12:40:17 - georgezgeorgez: but i think it would be good to say configure syrius to hook up to a nostr relay Mon, Mar 3, 2025, 12:40:24 - georgezgeorgez: which only sends out major updates Mon, Mar 3, 2025, 12:40:31 - coinselor: I can look into automating upgrade notifications, maybe looking at spork creation on-chain? Mon, Mar 3, 2025, 12:40:33 - georgezgeorgez: like "There's an upgrade available for Syrius" Mon, Mar 3, 2025, 12:40:51 - georgezgeorgez: sporks are really just for protocol upgrades Mon, Mar 3, 2025, 12:41:07 - georgezgeorgez: not necessarily great as a comm protocol Mon, Mar 3, 2025, 12:41:28 - deeznnutz: wonder if we could pull messages into syrius? Mon, Mar 3, 2025, 12:41:39 - georgezgeorgez: yes why not? if syrius had a nostr client into it Mon, Mar 3, 2025, 12:41:49 - georgezgeorgez: i'm thinking of how to get critical messages to users Mon, Mar 3, 2025, 12:42:05 - deeznnutz: that could actually be a good way to communicate with everyone Mon, Mar 3, 2025, 12:42:05 - georgezgeorgez: maybe most users won't check Telegram, matrix, or forums Mon, Mar 3, 2025, 12:42:14 - georgezgeorgez: but if they are opening wallet to check rewards Mon, Mar 3, 2025, 12:42:28 - georgezgeorgez: if they only see 1 important message there Mon, Mar 3, 2025, 12:42:54 - georgezgeorgez: definitely only notifications that likely require action Mon, Mar 3, 2025, 12:42:56 - coinselor: I think this should be automated:
- On github release -> post on all channels - On on-chain spork -> post on all channels etc Mon, Mar 3, 2025, 12:43:08 - georgezgeorgez: yes that's a good idea as well Mon, Mar 3, 2025, 12:43:15 - georgezgeorgez: could have some bots that monitor github and on-chain Mon, Mar 3, 2025, 12:43:17 - deeznnutz: ya with a GhA Mon, Mar 3, 2025, 12:43:56 - georgezgeorgez: but let's say just for HQZ for now Mon, Mar 3, 2025, 12:44:11 - georgezgeorgez: not all hqz pillars have a matrix or even a forum user iirc Mon, Mar 3, 2025, 12:44:20 - coinselor: and let users choose what the notifications they like best: x bot, tg bot, discord bot, matrix bot, etc Mon, Mar 3, 2025, 12:44:23 - georgezgeorgez: we are reaching them through AZ votes Mon, Mar 3, 2025, 12:44:25 - coinselor: nostr bot kek Mon, Mar 3, 2025, 12:44:29 - georgezgeorgez: but we can't do that forever Mon, Mar 3, 2025, 12:45:08 - georgezgeorgez: i'm thinking the minimum engagement is reward collecting Mon, Mar 3, 2025, 12:45:22 - georgezgeorgez: so if we can meet them there, we'll cast the widest net Mon, Mar 3, 2025, 12:45:24 - coinselor: show an ad on collect tab Mon, Mar 3, 2025, 12:45:25 - deeznnutz: is this something we should bring up in the Syrius SIG? maybe John M. would be interested in this Mon, Mar 3, 2025, 12:45:35 - deeznnutz: or CF Mon, Mar 3, 2025, 12:46:16 - georgezgeorgez: sure why not i think there will be some questions like should syrius be a generic nostr client? or should we have sort of protocol on top to indicate message priority Mon, Mar 3, 2025, 12:46:26 - coinselor: yeah I think syrius should prompt users whenever a new version is available Mon, Mar 3, 2025, 12:46:44 - georgezgeorgez: the question is how does syrius get that data Mon, Mar 3, 2025, 12:46:51 - georgezgeorgez: to know there's a new version Mon, Mar 3, 2025, 12:46:53 - coinselor: yeah I assume it's not trivial Mon, Mar 3, 2025, 12:47:00 - georgezgeorgez: centralized server? Mon, Mar 3, 2025, 12:47:07 - deeznnutz: wonder if that becomes an attack vector? Mon, Mar 3, 2025, 12:47:12 - coinselor: I've seen some crazy stuff for like new git versions with multiple devs signing stuff Mon, Mar 3, 2025, 12:47:23 - georgezgeorgez: well it's similar to our default rpc node selection right now Mon, Mar 3, 2025, 12:47:34 - georgezgeorgez: i think HC1 could run a nostr server Mon, Mar 3, 2025, 12:47:39 - georgezgeorgez: and we could set that as a default in syrius Mon, Mar 3, 2025, 12:47:51 - georgezgeorgez: but the option is there to change it Mon, Mar 3, 2025, 12:48:04 - georgezgeorgez: i mean with nostr, in theory you can subscribe to multiple relays Mon, Mar 3, 2025, 12:48:09 - coinselor: Technically we could have a frosted nostr multisig even to post messages Mon, Mar 3, 2025, 12:48:15 - georgezgeorgez: so i've been thinking about this and prototyping in nomctl Mon, Mar 3, 2025, 12:48:23 - georgezgeorgez: since for a while, that will be the hqz wallet Mon, Mar 3, 2025, 12:48:24 - deeznnutz: yep and we can hold a `.json` file that has nostr relays that people can submit PRs to add / remove them Mon, Mar 3, 2025, 12:48:28 - georgezgeorgez: and we need a fast way to tell hqz pillars Mon, Mar 3, 2025, 12:48:34 - georgezgeorgez: hey upgrade within a week please Mon, Mar 3, 2025, 12:48:54 - deeznnutz: cant remember, can we change the chainID in syrius now? Mon, Mar 3, 2025, 12:49:01 - deeznnutz: can we setup syrius to use HQZ? Mon, Mar 3, 2025, 12:49:14 - coinselor: I think so Mon, Mar 3, 2025, 12:49:29 - georgezgeorgez: i don't think we'll have most of the new functionality on syrius Mon, Mar 3, 2025, 12:49:35 - georgezgeorgez: until it's finalized on hqz Mon, Mar 3, 2025, 12:49:38 - georgezgeorgez: and ready for mainnet Mon, Mar 3, 2025, 12:49:53 - georgezgeorgez: i don't think we want to iterate over and over again with syrius Mon, Mar 3, 2025, 12:50:14 - georgezgeorgez: this is another reason i've been wanting syrius to have a "plugin" architecture Mon, Mar 3, 2025, 12:50:16 - georgezgeorgez: if possible Mon, Mar 3, 2025, 12:50:28 - georgezgeorgez: but it seems like from a Flutter perspective, it's not straightforward Mon, Mar 3, 2025, 12:50:43 - coinselor: yeah that makes no sense, but for stuff we haven't built like a notification system, we could iterate with hqz notifications and tweak the system until we are happy and then we just change node/chain id and it should work for nom too Mon, Mar 3, 2025, 12:51:01 - georgezgeorgez: like imagine you connect your wallet to a chain it detects all the supported features, and then configures the UI with the widgets that work for that chain Mon, Mar 3, 2025, 12:51:21 - deeznnutz: that would be cool Mon, Mar 3, 2025, 12:51:23 - georgezgeorgez: but long term dreams lol Mon, Mar 3, 2025, 12:51:54 - georgezgeorgez: for now, i'm just going to build out UI on nomctl and if the contracts are good on hqz then we can port it over to syrius Mon, Mar 3, 2025, 12:52:05 - georgezgeorgez: it's a lot easier to iterate UI on CLI/TUI Mon, Mar 3, 2025, 12:52:13 - coinselor: that seems doable, just check if the chain has liquidity embedded to show staking tab Mon, Mar 3, 2025, 12:53:05 - deeznnutz: I'm bumping up on my next meeting. I know what to do and will keep everyone informed as I make progress Mon, Mar 3, 2025, 12:53:06 - coinselor: nvm embeddeds have diff addresses on diff chains? Mon, Mar 3, 2025, 12:53:33 - georgezgeorgez: not necessarily, the addresses are the same on mainnet and hqz Mon, Mar 3, 2025, 12:54:22 - georgezgeorgez: lol, it's really just imagination talk right now Mon, Mar 3, 2025, 12:55:06 - georgezgeorgez: but for example each chain could have metadata like
here's a list of contracts: their addresses and some sort of UI specification for it Mon, Mar 3, 2025, 12:55:20 - georgezgeorgez: then the wallet can match up certain widgets to those different UI specifications Mon, Mar 3, 2025, 12:55:25 - coinselor: well I'm sure sooner rather than later we'll need an ez to setup local nom testnet. I know you've done it but it's not trivial for new devz i think Mon, Mar 3, 2025, 12:55:47 - georgezgeorgez: using ansible that should be fairly simple actually Mon, Mar 3, 2025, 12:56:23 - deeznnutz: i have one last question Mon, Mar 3, 2025, 12:56:34 - deeznnutz: what is the best way to make a wallet (producer) with ansible? Mon, Mar 3, 2025, 12:56:34 - georgezgeorgez: yes and we are hitting the end of the hour Mon, Mar 3, 2025, 12:56:48 - deeznnutz: dart cli tool and make a wallet called `producer` Mon, Mar 3, 2025, 12:56:49 - deeznnutz: ? Mon, Mar 3, 2025, 12:57:06 - georgezgeorgez: <@deeznnutz:zenon.chat "what is the best way to make a w..."> probably just call a CLI like dart tool or nomctl Mon, Mar 3, 2025, 12:57:30 - georgezgeorgez: i don't think we want ansible to worry about cryptographic logic Mon, Mar 3, 2025, 12:57:32 - deeznnutz: ya, so download the cli from GH and run the command to make the wallet Mon, Mar 3, 2025, 12:57:38 - georgezgeorgez: just install the cli then run the command Mon, Mar 3, 2025, 12:57:41 - georgezgeorgez: yah Mon, Mar 3, 2025, 12:57:48 - georgezgeorgez: and then after that, i think everything else is just templated files Mon, Mar 3, 2025, 12:57:50 - deeznnutz: ok - super easy Mon, Mar 3, 2025, 12:57:59 - georgezgeorgez: i think for nodes, there's also the network key Mon, Mar 3, 2025, 12:58:02 - coinselor: I should be back in full form in a few days, I'll help if you can create/list some pending tasks ^^ Mon, Mar 3, 2025, 12:58:23 - georgezgeorgez: <@georgezgeorgez:hc1.chat "i think for nodes, there's also ..."> maybe for nomctl i can create a utility command which just creates that Mon, Mar 3, 2025, 12:58:36 - georgezgeorgez: this way we can setup the entire file structure without having to start up the node first Mon, Mar 3, 2025, 12:58:48 - deeznnutz: ya that would be helpful for sure Mon, Mar 3, 2025, 12:59:03 - deeznnutz: what is the network key? Mon, Mar 3, 2025, 12:59:07 - deeznnutz: not familiar with that Mon, Mar 3, 2025, 12:59:35 - coinselor: i think he means the node id ? Mon, Mar 3, 2025, 12:59:37 - georgezgeorgez: it's the key that is used for the enode i think in the folder it's literally a file called like network-key Mon, Mar 3, 2025, 12:59:56 - georgezgeorgez: so there's a producer key, which signs created transactions Mon, Mar 3, 2025, 13:00:03 - deeznnutz: Something that Pillars need? Mon, Mar 3, 2025, 13:00:12 - georgezgeorgez: and then each node has a key used for the p2p network which yeah, gets represented in node id Mon, Mar 3, 2025, 13:00:24 - deeznnutz: I did not know that Mon, Mar 3, 2025, 13:00:27 - georgezgeorgez: all nodes create one when they start up if it isn't there i think Mon, Mar 3, 2025, 13:00:30 - deeznnutz: I'll look into it Mon, Mar 3, 2025, 13:00:43 - georgezgeorgez: it's used for example to verify you haven't been man in the middled Mon, Mar 3, 2025, 13:00:45 - coinselor: yeah supernova has something similar i think Mon, Mar 3, 2025, 13:00:53 - georgezgeorgez: and that the node you're talking to on this IP Mon, Mar 3, 2025, 13:00:56 - georgezgeorgez: is actually that one Mon, Mar 3, 2025, 13:01:06 - deeznnutz: I see Mon, Mar 3, 2025, 13:01:17 - deeznnutz: prevents spoofing the IP Mon, Mar 3, 2025, 13:01:37 - georgezgeorgez: yeah stuff like that would have to go deep into the p2p stack to cover all its functionality Mon, Mar 3, 2025, 13:01:49 - georgezgeorgez: and once we switch to libp2p that file might go away/change Mon, Mar 3, 2025, 13:02:06 - georgezgeorgez: but first major upgrade functionality focused Mon, Mar 3, 2025, 13:02:19 - deeznnutz: cool. I'm all set. Mon, Mar 3, 2025, 13:02:32 - georgezgeorgez: awesome. thanks everyone good meeting Mon, Mar 3, 2025, 13:02:47 - coinselor: thank you all Mon, Mar 3, 2025, 13:02:49 - deeznnutz: thanks everyone.!!!! great meeting Mon, Mar 3, 2025, 13:02:57 - deeznnutz: ==== OP SIG END ====