Paths

Utilities to manipulate Paths

print(P().ls())
print(P().resolve())

[» _quarto.yml, » decorators.ipynb, » markups.ipynb, » sidebar.yml, » AttrDict.ipynb, » interactive_show.ipynb, » load_defautls.ipynb, » sklegos.ipynb, » bounding_boxes.ipynb, » show.ipynb, » pdf.ipynb, » charts.ipynb, » paths.ipynb, » nbdev.yml, » tmp.csv, » jupyter_notebook.ipynb, » config.ipynb, » misc.ipynb, » registry.ipynb, » adapters.ipynb, » report.ipynb, » .ipynb_checkpoints, » capsule.ipynb, » logging.ipynb, » inspector.ipynb, » bokeh_plotting.ipynb, » index.ipynb, » imgaug_loader.ipynb]
/Users/yeshwanth/Code/Personal/torch_snippets/nbs

# !touch tmp.txt tmp.csv
# # x = P()
# x.tmp__csv, x.tmp__txt, x.misc

!rm tmp.txt tmp.misc

rm: tmp.txt: No such file or directory
rm: tmp.misc: No such file or directory

---

Path.rm

 Path.rm (confirm_prompt='Are you sure you want to delete `{self}`?
          [y/N]', silent=True, missing_ok=True, force=False,
          dry_run=False)

---

remove_file

 remove_file (dry_run)

---

Path.cp

 Path.cp (to)

---

Path.mv

 Path.mv (to)

---

Path.sample

 Path.sample (pattern='*')

---

extn

 extn (fpath)

---

Path.extn

 Path.extn ()

---

Path.sz

 Path.sz ()

---

Path.size

 Path.size ()

---

Path.rmtree

 Path.rmtree (prompt='Really remove `{self}` and its contents? [y/n] ',
              force=False)

p = P("test.txt")
p.touch()
logger.info(p.size())

[08/24/24 11:12:12] INFO     0 KB                                                                                                                   1417366103.py:<module>:3

Path objects can be moved and copied

p = p.mv("test1.txt")
q = p.cp("test2.txt")

Path objects have a size, extn (extension) and parent attributes

assert isinstance(q, P)
assert q.size() == "0 KB"
assert str(p) == "test1.txt"
assert p.extn == "txt"
assert p.parent == P()

They can be deleted with/without a prompt

p.rm(force=True)
q.rm(confirm_prompt=False)

Folders can be globbed with a default of everything

p = P("../torch_snippets")
assert P().ls() == P().Glob()
p.Glob("*.py")

(#34) [» ../torch_snippets/misc.py,» ../torch_snippets/load_defaults.py,» ../torch_snippets/text_utils.py,» ../torch_snippets/_nbdev.py,» ../torch_snippets/paths.py,» ../torch_snippets/charts.py,» ../torch_snippets/pdf_loader.py,» ../torch_snippets/interactive_show.py,» ../torch_snippets/registry.py,» ../torch_snippets/markup2.py,» ../torch_snippets/_modidx.py,» ../torch_snippets/inspector.py,» ../torch_snippets/__init__.py,» ../torch_snippets/tmp.py,» ../torch_snippets/torch_loader.py,» ../torch_snippets/logger.py,» ../torch_snippets/markup.py,» ../torch_snippets/fastcores.py,» ../torch_snippets/sklegos.py,» ../torch_snippets/cli.py...]

You can sample a random file from the directory

q = p.sample("*.py")
Info(f"Sample file: `{q}`")
Info(f"Sample file size: `{q.size()}`")

                    INFO     Sample file: `../torch_snippets/logger.py`                                                                             1725593160.py:<module>:2

                    INFO     Sample file size: `9 KB`                                                                                               1725593160.py:<module>:3

try:
    p.size()
except Exception as e:
    logger.warning(e)

                    WARNING  `../torch_snippets` is a directory                                                                                     1980994904.py:<module>:4

p = P("test.txt")
p.touch()
assert isdir(p) == False
assert fname(p) == "test.txt"
assert parent(p) == P()
assert stem(p) == "test"
assert extn(p) == "txt"

print(find("capsule", Glob("./")))

p.rm(confirm_prompt=False)

capsule.ipynb

---

list_zip

 list_zip (file)

---

unzip_file

 unzip_file (file, dest)

---

zip_files

 zip_files (list_of_files, dest)

!touch test1.txt test2.txt
f = zip_files(P().Glob("*.txt"), "test.tar.gz")
unzip_file(f, "./")
[f.rm(force=True) for f in P().Glob("*.txt")]
P("test.tar.gz").rm(force=True)

!touch test1.txt test2.txt
f = zip_files(P().Glob("*.txt"), "test.zip")
unzip_file(f, "./")
[f.rm(force=True) for f in P().Glob("*.txt")]
P("test.zip").rm(force=True)

                    INFO     Zipping 2 files to test.tar.gz...                                                                                     3826617683.py:zip_files:8

100%|██████████| 2/2 [00:00<00:00, 649.78it/s]
/var/folders/1_/71dqv9vx2750gmyz77q_f45w0000gn/T/ipykernel_40147/3826617683.py:27: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
  f.extractall(dest)

                    INFO     Zipping 2 files to test.zip...                                                                                        3826617683.py:zip_files:8

100%|██████████| 2/2 [00:00<00:00, 2958.94it/s]

---

folder_summary

 folder_summary (thing)

---

common_items

 common_items (*fldrs, verbose=True)

---

remove_duplicates

 remove_duplicates (files)

Check a list of files and remove duplicates based on their checksum

---

md5

 md5 (fname)

md5("paths.ipynb")

'74eb392fa44e9293fb30ffed68c14112'

---

Path.write_lines

 Path.write_lines (lines, mode)

---

writelines

 writelines (lines, file, mode)

---

Path.read_file

 Path.read_file (**kwargs)

---

readfile

 readfile (*args, **kwargs)

---

Path.read_lines

 Path.read_lines (silent=False, encoding=None)

---

readlines

 readlines (fpath, silent=False, encoding=None, _strip=True)

P("paths.ipynb").read_lines()[:10]

                    INFO     loaded 1075 lines                                                                                                   2901855681.py:read_lines:13

['{',
 '"cells": [',
 '{',
 '"cell_type": "raw",',
 '"metadata": {},',
 '"source": [',
 '"# Paths\\n",',
 '"Utilities to manipulate Paths\\n",',
 '"---\\n",',
 '"{}\\n",']

---

tree

 tree (directory='./', filelimit=50, to=None)

P("../").tree

/Users/yeshwanth/Code/Personal/torch_snippets
 ── LICENSE
 ── LICENSE.txt
 ── MANIFEST
 ── MANIFEST.in
 ── Makefile
 ── README.md
 ── Screenshot 2022-09-08 at 4.40.18 PM.png
 ── __module_timing__.py.lprof
 ── _proc
     ── AttrDict.ipynb
     ── _quarto.yml
     ── adapters.ipynb
     ── bokeh_plotting.ipynb
     ── bounding_boxes.ipynb
     ── capsule.ipynb
     ── charts.ipynb
     ── config.ipynb
     ── decorators.ipynb
     ── docs
         ── AttrDict.html
         ── adapters.html
         ── bokeh_plotting.html
         ── bounding_boxes.html
         ── bounding_boxes_files
             ── figure-html
                 ── cell-6-output-1.png
                 ── cell-7-output-1.png
                 ── cell-8-output-1.png
                 ── cell-9-output-1.png
         ── capsule.html
         ── capsule_files
             ── figure-html
                 ── cell-6-output-2.png
         ── charts.html
         ── charts_files
             ── figure-html
                 ── cell-7-output-1.png
         ── config.html
         ── decorators.html
         ── imgaug_loader.html
         ── imgaug_loader_files
             ── figure-html
                 ── cell-3-output-1.png
                 ── cell-5-output-10.png
                 ── cell-5-output-12.png
                 ── cell-5-output-14.png
                 ── cell-5-output-16.png
                 ── cell-5-output-18.png
                 ── cell-5-output-2.png
                 ── cell-5-output-20.png
                 ── cell-5-output-22.png
                 ── cell-5-output-4.png
                 ── cell-5-output-6.png
                 ── cell-5-output-8.png
                 ── cell-6-output-1.png
                 ── cell-7-output-1.png
                 ── cell-8-output-2.png
         ── index.html
         ── inspector.html
         ── interactive_show.html
         ── jupyter_notebook.html
         ── load_defautls.html
         ── logging.html
         ── markups.html
         ── misc.html
         ── paths.html
         ── pdf.html
         ── registry.html
         ── report.html
         ── report_files
             ── figure-html
                 ── cell-3-output-2.png
                 ── cell-5-output-1.png
         ── robots.txt
         ── search.json
         ── show.html
         ── show_files
             ── figure-html
                 ── cell-2-output-1.png
                 ── cell-3-output-1.png
                 ── cell-4-output-2.png
                 ── cell-4-output-4.png
                 ── cell-4-output-6.png
                 ── cell-5-output-1.png
                 ── cell-8-output-1.png
                 ── cell-9-output-1.png
         ── site_libs
             ── bootstrap
                 ── bootstrap-icons.css
                 ── bootstrap-icons.woff
                 ── bootstrap.min.css
                 ── bootstrap.min.js
             ── clipboard
                 ── clipboard.min.js
             ── quarto-html
                 ── anchor.min.js
                 ── popper.min.js
                 ── quarto-syntax-highlighting.css
                 ── quarto.js
                 ── tippy.css
                 ── tippy.umd.min.js
             ── quarto-nav
                 ── headroom.min.js
                 ── quarto-nav.js
             ── quarto-search
                 ── autocomplete.umd.js
                 ── fuse.min.js
                 ── quarto-search.js
         ── sitemap.xml
         ── sklegos.html
     ── imgaug_loader.ipynb
     ── index.ipynb
     ── inspector.ipynb
     ── interactive_show.ipynb
     ── jupyter_notebook.ipynb
     ── load_defautls.ipynb
     ── logging.ipynb
     ── markups.ipynb
     ── misc.ipynb
     ── nbdev.yml
     ── paths.ipynb
     ── pdf.ipynb
     ── registry.ipynb
     ── report.ipynb
     ── show.ipynb
     ── sidebar.yml
     ── sklegos.ipynb
     ── test.txt
     ── test1.txt
     ── test2.txt
     ── tmp.csv
     ── tmp.txt
 ── assets
     ── Preamble.csv
     ── Preamble.png
     ── avgs0.png
     ── avgs1.png
     ── demo.gif
 ── backups
     ── testing
         ── 0000.html
 ── build
     ── bdist.macosx-11.1-arm64
     ── lib
         ── torch_snippets
             ── __init__.py
             ── __module_timing__.py
             ── _modidx.py
             ── _nbdev.py
             ── adapters.py
             ── bb_utils.py
             ── bokeh_loader.py
             ── charts.py
             ── cli.py
             ── dates.py
             ── decorators.py
             ── fastcores.py
             ── icecream.py
             ── imgaug_loader.py
             ── inspector.py
             ── interactive_show.py
             ── ipython.py
             ── load_defaults.py
             ── loader.py
             ── logger.py
             ── markup.py
             ── markup2.py
             ── misc.py
             ── paths.py
             ── pdf_loader.py
             ── registry.py
             ── s3_loader.py
             ── scp.py
             ── sklegos.py
             ── text_utils.py
             ── thinc_parser
                 ── __init__.py
                 ── parser.py
             ── tmp.py
             ── torch_loader.py
             ── trainer
                 ── __init__.py
                 ── capsule.py
                 ── config.py
                 ── hooks.py
                 ── neural_graph.py
             ── video.py
             ── zen.py
 ── changelog.md
 ── conda
     ── torch_snippets
         ── meta.yaml
 ── debug
     ── profile_time.txt
     ── profile_time_202408241027.txt
 ── dist
     ── torch_snippets-0.6.0-py3-none-any.whl
     ── torch_snippets-0.6.0.tar.gz
 ── docker-compose.yml
 ── docs
     ── AttrDict.html
     ── adapters.html
     ── bokeh_plotting.html
     ── bounding_boxes.html
     ── bounding_boxes_files
         ── figure-html
             ── cell-6-output-1.png
             ── cell-7-output-1.png
             ── cell-8-output-1.png
             ── cell-9-output-1.png
     ── capsule.html
     ── capsule_files
         ── figure-html
             ── cell-5-output-2.png
     ── charts.html
     ── charts_files
         ── figure-html
             ── cell-7-output-1.png
     ── config.html
     ── decorators.html
     ── imgaug_loader.html
     ── imgaug_loader_files
         ── figure-html
             ── cell-3-output-1.png
             ── cell-5-output-10.png
             ── cell-5-output-12.png
             ── cell-5-output-14.png
             ── cell-5-output-16.png
             ── cell-5-output-18.png
             ── cell-5-output-2.png
             ── cell-5-output-20.png
             ── cell-5-output-22.png
             ── cell-5-output-4.png
             ── cell-5-output-6.png
             ── cell-5-output-8.png
             ── cell-6-output-1.png
             ── cell-7-output-1.png
             ── cell-8-output-2.png
     ── index.html
     ── inspector.html
     ── interactive_show.html
     ── jupyter_notebook.html
     ── load_defautls.html
     ── logging.html
     ── markups.html
     ── misc.html
     ── paths.html
     ── pdf.html
     ── registry.html
     ── report.html
     ── report_files
         ── figure-html
             ── cell-3-output-2.png
             ── cell-5-output-1.png
     ── robots.txt
     ── search.json
     ── show.html
     ── show_files
         ── figure-html
             ── cell-2-output-1.png
             ── cell-3-output-1.png
             ── cell-4-output-2.png
             ── cell-4-output-4.png
             ── cell-4-output-6.png
             ── cell-5-output-1.png
             ── cell-8-output-1.png
             ── cell-9-output-1.png
     ── site_libs
         ── bootstrap
             ── bootstrap-icons.css
             ── bootstrap-icons.woff
             ── bootstrap.min.css
             ── bootstrap.min.js
         ── clipboard
             ── clipboard.min.js
         ── quarto-html
             ── anchor.min.js
             ── popper.min.js
             ── quarto-syntax-highlighting.css
             ── quarto.js
             ── tippy.css
             ── tippy.umd.min.js
         ── quarto-nav
             ── headroom.min.js
             ── quarto-nav.js
         ── quarto-search
             ── autocomplete.umd.js
             ── fuse.min.js
             ── quarto-search.js
     ── sitemap.xml
     ── sklegos.html
 ── nbs
     ── AttrDict.ipynb
     ── _quarto.yml
     ── adapters.ipynb
     ── bokeh_plotting.ipynb
     ── bounding_boxes.ipynb
     ── capsule.ipynb
     ── charts.ipynb
     ── config.ipynb
     ── decorators.ipynb
     ── imgaug_loader.ipynb
     ── index.ipynb
     ── inspector.ipynb
     ── interactive_show.ipynb
     ── jupyter_notebook.ipynb
     ── load_defautls.ipynb
     ── logging.ipynb
     ── markups.ipynb
     ── misc.ipynb
     ── nbdev.yml
     ── paths.ipynb
     ── pdf.ipynb
     ── registry.ipynb
     ── report.ipynb
     ── show.ipynb
     ── sidebar.yml
     ── sklegos.ipynb
     ── tmp.csv
 ── requirements.txt
 ── scripts.ipynb
 ── settings.ini
 ── setup.cfg
 ── setup.py
 ── testing.ipynb
 ── tmp.ini
 ── torch_snippets
     ── __init__.py
     ── __module_timing__.py
     ── __pycache__
         ── __init__.cpython-312.pyc
         ── __module_timing__.cpython-312.pyc
         ── bb_utils.cpython-312.pyc
         ── bokeh_loader.cpython-312.pyc
         ── charts.cpython-312.pyc
         ── cli.cpython-312.pyc
         ── dates.cpython-312.pyc
         ── decorators.cpython-312.pyc
         ── icecream.cpython-312.pyc
         ── inspector.cpython-312.pyc
         ── ipython.cpython-312.pyc
         ── load_defaults.cpython-312.pyc
         ── loader.cpython-312.pyc
         ── logger.cpython-312.pyc
         ── markup.cpython-312.pyc
         ── markup2.cpython-312.pyc
         ── misc.cpython-312.pyc
         ── paths.cpython-312.pyc
         ── pdf_loader.cpython-312.pyc
         ── registry.cpython-312.pyc
         ── s3_loader.cpython-312.pyc
         ── torch_loader.cpython-312.pyc
         ── zen.cpython-312.pyc
     ── _modidx.py
     ── _nbdev.py
     ── adapters.py
     ── bb_utils.py
     ── bokeh_loader.py
     ── charts.py
     ── cli.py
     ── dates.py
     ── decorators.py
     ── fastcores.py
     ── icecream.py
     ── imgaug_loader.py
     ── inspector.py
     ── interactive_show.py
     ── ipython.py
     ── load_defaults.py
     ── loader.py
     ── logger.py
     ── markup.py
     ── markup2.py
     ── misc.py
     ── paths.py
     ── pdf_loader.py
     ── registry.py
     ── s3_loader.py
     ── scp.py
     ── sklegos.py
     ── text_utils.py
     ── thinc_parser
         ── __init__.py
         ── __pycache__
             ── __init__.cpython-312.pyc
             ── parser.cpython-312.pyc
         ── parser.py
     ── tmp.py
     ── torch_loader.py
     ── trainer
         ── __init__.py
         ── __pycache__
             ── __init__.cpython-312.pyc
             ── capsule.cpython-312.pyc
         ── capsule.py
         ── config.py
         ── hooks.py
         ── neural_graph.py
     ── video.py
     ── zen.py
 ── torch_snippets.egg-info
     ── PKG-INFO
     ── SOURCES.txt
     ── dependency_links.txt
     ── entry_points.txt
     ── not-zip-safe
     ── requires.txt
     ── top_level.txt

61 directories, 348 files

---

folder_structure_to_json

 folder_structure_to_json (path, output_file=None)

Creates a JSON file representing the folder structure of the given directory.

---

folder_structure_to_dict

 folder_structure_to_dict (path)

Recursively constructs a nested dictionary that represents the folder structure.

x = P("tmp.txt")
x.touch()
x.write_lines([i for i in range(1000)], mode="w")
lines = x.read_lines()
assert lines == [f"{i}" for i in range(1000)]
logger.info(x.size())

x.rm(confirm_prompt=False)

[08/24/24 11:12:13] INFO     loaded 1000 lines                                                                                                   2901855681.py:read_lines:13

                    INFO     3 KB                                                                                                                   1033976698.py:<module>:6

---

loaddill

 loaddill (fpath)

Load a python object from a dill file

---

dumpdill

 dumpdill (obj, fpath, silent=False, message='Dumped object of size
           ≈{fsize} @ "{fpath}" in {dumptime:.2e} seconds')

Dump a python object as a dill file (better replacement to pickle)

p = P("test.tmp")
dumpdill([1, 2, 3], p)
y = loaddill(p)
p.rm(confirm_prompt=False)
assert y == [1, 2, 3]

                    INFO     Dumped object of size ≈0 KB @ "test.tmp" in 4.11e-04 seconds                                                            554288780.py:<module>:2