1

I'm working on a small python project and have a general_code.py file where I have functions that I use throughout the project. Some examples: 

def to_boolean(var):
    return var in ['True', 'true', '1', 'y', 'Y' 'yes', 'Yes']

e.g 2

def get_last_line(in_file, block_size=128, ignore_ending_newline=False):

e.g 3

def signal_handler(log, self, signum, frame):  # pragma: no cover
    log.info("Ctrl + C pressed. Exiting application gracefully")
    self.run = False
    while threading.active_count() > 1:
        time.sleep(1)
    log.info("All threads closed. Exit.")
    sys.exit(0)

e.g 4.

class Timer(object):  # pragma: no cover
    def __init__(self, name=None):
        self.name = name

    def __enter__(self):
        self.tstart = time.time()

    def __exit__(self, type, value, traceback):
        if self.name:
            print('[%s]' % self.name,)
        print('Elapsed: %s' % (time.time() - self.tstart))

e.g 5

def get_file_version(log, filename=None):

e.g 6

def setup_logger(filename):

Is it good design to have such a central place for common functionality? It has served me well, but often ideas that might be great in a small scope can end up terrible in bigger projects or different scopes. So what would be a good way in python to group common functionality? Is having all functions simply in one file like I do fine or would it be better to put them all into an own class or something completely different?

Hakaishin
  • 197
  • 7

2 Answers2

6

It can be fine for smaller use-cases, though it's not ideal. If you have 3 or so utility functions it's manageable in your head to keep them in 1 file, but it quickly gets out of hand and it's roughly impossible to navigate a god-file with 200 utility functions where some are related and others are not (too much cognitive load).

I'd definitely shy away from making one "general code" file: the name of the import doesn't give any information on what it's doing, and as a project grows it has the potential to be a file that's imported in everything (and due to being used everywhere, they may get extended until you have a dozen optional flags on every function so that you can get returns just right).

As a solution that helps to manage that, I'd suggest moving grouped functionality into their own files. So you might have a "file_utils.py" file that contains exapmles 2 and 5, a "debug_utils.py" file which contains 4 and 6, "signal_utils.py" which contains 3, and an "input_parsing_utils.py" which contains 1. Each of these is then nicely packaged bundles of functionality so you can have exactly the functionality you need and are nicely extensible. More generally, you want to get groups of functionality that are closely related put into actual groups, and you want to keep anything unrelated to those functions completely out of the way (out of sight, out of mind).

As an added bonus, the sets of functionality are separate - so if you end up needing a ton of flags for a function you can more easily extract that into a factory or strategy pattern.

Ideally, these utility files should be totally independent from one another (a util shouldn't need another util to work - if it does, it probably isn't a util). Packaging up discrete bundles of functionality that can be extended or refactored without needing to touch any other unrelated functionality is a good thing. It's also notable that things should only be utilities if they have the potential to be reused - if you write a function that does one thing that's very specific to an internal of the system and needs to return values in a very specific format, it's probably not worth making that a utility (just a private-ish function on the class that needs it).

Delioth
  • 433
  • 4
  • 8
  • Exactly. "General code" is not a bad idea, but definitely bad naming. If you can't name a thing well, it's usually because you fail to understand its purpose well. Fixing the latter will help. – 9000 Sep 11 '18 at 15:01
  • Ok I see, so it depends on project scope. If it is small it is fine to keep things in one place, but when the project and the file grows it's better to group up similar things. I actually already did this once, where I factored a few functions out into a separate file. The functions are self contained so that was and will be no problem. Thanks for the nice answer. – Hakaishin Sep 11 '18 at 15:03
  • @Hakaishin It's less of "okay to do in small projects" and more the idea that it's still manageable in small projects. If you only have 3 utility functions it's still a good idea to put them in their own grouped files (which may only contain 1 function), but it's easy to refactor and manageable in your head. That simply becomes impossible when you have a few hundred utility functions. I'd suggest always keeping things in their own named files, regardless of project scope (because scope can always grow). – Delioth Sep 11 '18 at 15:06
  • Hmm ok, I guess it would be cleaner and the only "disadvantage" would be having multiple files, with the advantage of better naming. – Hakaishin Sep 11 '18 at 15:10
  • @Hakaishin Exactly - and having multiple files isn't really a disadvantage. Any reasonable IDE will find those files and be able to auto-import from them when you need them or switch between them at a moment's notice, and imports are usually the last thing one should worry about (for performance, they're negligible compared to the overhead of Python, in compiled languages they're pretty much only notable at compile-time). – Delioth Sep 11 '18 at 15:13
  • Could you maybe incorporate your comments into the answer? You get the point across well with examples, but maybe something along the lines consider splitting up general_code into more descriptive units and group those then into separate files – Hakaishin Sep 11 '18 at 15:20
1

Having common code files isn't a bad idea, but it is a dangerous one. It's not an ideal solution, but it is so common and simple that it is heavily used in most real projects. A lot of discipline is required to prevent common files from doing too much, becoming god classes and making your code much more confusing. A common file should be a home of last resort for functions, and it should never have functions doing business logic. Another common pitfall is making functions in a common file that aren't actually common, this is generally due to trying to reduce duplication aggressively. This generally manifests in having function like: foo(); foo2(); foo3(); or having one function with many parameters that are mostly optional. If you stay disciplined in your coding then a common code file can be a great tool, but it can become a nightmare if you aren't careful.

Ryathal
  • 13,317
  • 1
  • 33
  • 48
  • 1
    Could you maybe elaborate why it should be a last resort? I see that there shouldn't be too much in there, but what better place would there be for eg. def get_file_version(log, filename=None): or def setup_logger(): ? It's own get_file_version file/class or logging class? – Hakaishin Sep 11 '18 at 13:12
  • 1
    @Hakaishin see this question: https://softwareengineering.stackexchange.com/q/339276/20756 – Blrfl Sep 11 '18 at 16:37
  • Great read, especially the third answer is very good – Hakaishin Sep 11 '18 at 17:01