8:58 a.m.
Hi!
I'm a member of a team on storage which is trying to make storage more accessible to
consumers.
Yesterday, there was a brief dicussion in the Project Springfield meeting of pyblk,
which is a sort of lsblk-like library.
We believe that the capabilities of pyblk would enhance the smart storage
administrator's
abilities a lot.
I'm looking for feedback on how Cockpit might be able to use these capabilities, and
what Cockpit would require to make these capabilities accessible to it.
If you can, could you take a look at the whitepaper here:
https://mulhern.fedorapeople.org/pyblk_rfc.pdf ?
We have discussed a lot of ways the info pyblk can generate could be served up to the
user. For example, we could attach graphs (or pointers to the graphs) to journal
entries. We could be more independent and instead have a facility to look up graphs by
means of time-stamps. We can present diffs of graphs of the state of storage before
and after a certain time.
What do you think of any of this? Please let us know.
Thanks!
- mulhern
1:59 a.m.
Anne Mulhern <amulhern(a)redhat.com> writes:
I'm looking for feedback on how Cockpit might be able to use
these
capabilities, and what Cockpit would require to make these
capabilities accessible to it.
If you can, could you take a look at the whitepaper here:
https://mulhern.fedorapeople.org/pyblk_rfc.pdf ?
We have discussed a lot of ways the info pyblk can generate could be
served up to the user. For example, we could attach graphs (or
pointers to the graphs) to journal entries. We could be more
independent and instead have a facility to look up graphs by means of
time-stamps. We can present diffs of graphs of the state of storage
before and after a certain time.
What do you think of any of this? Please let us know.
So, pyblk offers two new things, right?
- Better display of the storage 'hierarchy' by admitting that it is
actually a DAG and that nodes can have more than one parent
- Facilities for snapshotting configurations over time and comparing
them
Correct? Did I miss something big?
Cockpit does quite a bit of storage hierarchy walking itself, and
currently it uses unsatisfying code full of case analysis. It would be
nice to generalize this.
Of course, there are dangers in generalizing, and Cockpit should always
strive to natively understand all the different kinds of storage
objects. However, showing a generic DAG is better than just showing
unknown things as "Other Devices". The code will be cleaner even for
the things Cockpit does know about.
Also, while Cockpit does allow navigation of the storage hierarchy, it
doesn't provide a good overview of it. That would be a valuable
addition.
The way to make the pyblk features accessible to Cockpit would be to
integrate pyblk with Storaged:
https://github.com/storaged-project/storaged[1]
It might start out as a module that adds new StorageGraph interfaces to
all relevant objects, with Parents and Children properties.
Investigating the history of storage configurations could probably be
added quite easily to the overview page. I think Storaged would also be
the right place to expose this to Cockpit.
As a more detailed comment, I think that looking at the device mapper
nodes is not the right level for LVM. Instead, we should look at the
'native' LVM objects, physical volumes, volume groups, pools, snapshots,
and logical volumes. This means that pyblk needs to know about LVM, but
if it doesn't, then it is much less useful for Cockpit, unfortunately.
3:44 p.m.
New subject: Request for comment and suggestions on pyblk and Cockpit
integration
----- Original Message -----
From: "Marius Vollmer" <marius.vollmer(a)redhat.com>
To: "Development discussion for the Cockpit Project"
<cockpit-devel(a)lists.fedorahosted.org>
Sent: Tuesday, November 17, 2015 2:59:51 AM
Subject: Re: Request for comment and suggestions on pyblk and Cockpit integration
Anne Mulhern <amulhern(a)redhat.com> writes:
> I'm looking for feedback on how Cockpit might be able to use these
> capabilities, and what Cockpit would require to make these
> capabilities accessible to it.
>
> If you can, could you take a look at the whitepaper here:
> https://mulhern.fedorapeople.org/pyblk_rfc.pdf ?
>
> We have discussed a lot of ways the info pyblk can generate could be
> served up to the user. For example, we could attach graphs (or
> pointers to the graphs) to journal entries. We could be more
> independent and instead have a facility to look up graphs by means of
> time-stamps. We can present diffs of graphs of the state of storage
> before and after a certain time.
>
> What do you think of any of this? Please let us know.
So, pyblk offers two new things, right?
- Better display of the storage 'hierarchy' by admitting that it is
actually a DAG and that nodes can have more than one parent
- Facilities for snapshotting configurations over time and comparing
them
Correct? Did I miss something big?
Yes, that's right, with the emphasis that it is not just display but comparison,
storage,
_everything_ made more sensible by using a DAG representation.
One thing that I subsequently realized that I failed to mention in the
document, is that some of pyblk's expressiveness comes from the powerful
networkx library that allows it to calculate graph isomorphisms in
a very flexible way. So, the graph comparisons can be made quite interesting,
but in large part because of this nice library (which in turns depends
to some extent on a bunch of C libraries).
Cockpit does quite a bit of storage hierarchy walking itself, and
currently it uses unsatisfying code full of case analysis. It would be
nice to generalize this.
Of course, there are dangers in generalizing, and Cockpit should always
strive to natively understand all the different kinds of storage
objects. However, showing a generic DAG is better than just showing
unknown things as "Other Devices". The code will be cleaner even for
the things Cockpit does know about.
Also, while Cockpit does allow navigation of the storage hierarchy, it
doesn't provide a good overview of it. That would be a valuable
addition.
The way to make the pyblk features accessible to Cockpit would be to
integrate pyblk with Storaged:
https://github.com/storaged-project/storaged[1]
It might start out as a module that adds new StorageGraph interfaces to
all relevant objects, with Parents and Children properties.
I can take a more in-depth look at storaged and see how this would work.
If I have questions should I ask you or ....?
Investigating the history of storage configurations could probably be
added quite easily to the overview page. I think Storaged would also be
the right place to expose this to Cockpit.
As a more detailed comment, I think that looking at the device mapper
nodes is not the right level for LVM. Instead, we should look at the
'native' LVM objects, physical volumes, volume groups, pools, snapshots,
and logical volumes. This means that pyblk needs to know about LVM, but
if it doesn't, then it is much less useful for Cockpit, unfortunately.
This could be more or less complex. The current names shown are just
udev's DEVNAME and are not so informative, everybody agrees. Showing for
lvm the particularly names that dm uses would be absolutely straightforward.
But taking a plunge into showing whatever more complex relationships lvm
builds on top of dm would be another step. Currently, all operations rely
solely on information that can be extracted directly from sysfs or the
udev database.
_______________________________________________
cockpit-devel mailing list
cockpit-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/admin/lists/cockpit-devel@lists.fedorahost...
- mulhern
7:04 a.m.
New subject: Request for comment and suggestions on pyblk and Cockpit
integration
<--SNIP -->
> As a more detailed comment, I think that looking at the device
mapper
> nodes is not the right level for LVM. Instead, we should look at the
> 'native' LVM objects, physical volumes, volume groups, pools, snapshots,
> and logical volumes. This means that pyblk needs to know about LVM, but
> if it doesn't, then it is much less useful for Cockpit, unfortunately.
This could be more or less complex. The current names shown are just
udev's DEVNAME and are not so informative, everybody agrees. Showing for
lvm the particularly names that dm uses would be absolutely straightforward.
But taking a plunge into showing whatever more complex relationships lvm
builds on top of dm would be another step. Currently, all operations rely
solely on information that can be extracted directly from sysfs or the
udev database.
So the coding for this change really did take just five minutes
(https://github.com/mulkieran/pyblk/commit/03acb3d60369514e9b98676eaa2798d...).
And the new results look something like this.
DEVNAME DMNAME DEVTYPE
DIFFSTATUS SIZE
/dev/nvme0n1 None disk None
@372.61 GiB
`-/dev/md127 None disk None
@372.49 GiB
|-/dev/dm-29 vg-cache0_cmeta disk None
16.00 MiB
`-/dev/dm-31 vg-lv_tdata disk None
32.00 GiB
`-/dev/dm-32 vg-lv-tpool disk None
32.00 GiB
|-/dev/dm-33 vg-lv disk None
32.00 GiB
`-/dev/dm-34 vg-thin1 disk None
1.00 TiB
...
Is that what you were getting at?
If you wanted to have the DMNAME override the DEVNAME whenever it was available that would
take another five minutes.
(I like to know where my data is coming from, so keeping them separate suits me).
- mulhern
>
>
> > _______________________________________________
> > cockpit-devel mailing list
> > cockpit-devel(a)lists.fedorahosted.org
> > https://lists.fedorahosted.org/admin/lists/cockpit-devel@lists.fedorahost...
> >
>
> - mulhern
> _______________________________________________
> cockpit-devel mailing list
> cockpit-devel(a)lists.fedorahosted.org
> https://lists.fedorahosted.org/admin/lists/cockpit-devel@lists.fedorahost...
>
3074
days inactive
3080
days old
cockpit-devel@lists.fedorahosted.org
3 comments
2 participants
participants (2)
-
Anne Mulhern -
Marius Vollmer