Ways to remove duplicates from list in Python

In Python, there are several ways to remove duplicates from a list. Below are some common methods:

1. Using a Set

A set is a data structure in Python that does not allow duplicate values. By converting a list into a set, duplicates are automatically removed. However, the order of the elements may not be preserved.

Example:

# Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Convert to set to remove duplicates
unique_list = list(set(my_list))

print(unique_list)

Output:

[1, 2, 3, 4, 5]

Note: The order of elements may not be the same as the original list.

2. Using a Loop and a New List

You can iterate through the list and add each item to a new list only if it is not already present.

Example:

# Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Create a new list and append only unique items
unique_list = []
for item in my_list:
    if item not in unique_list:
        unique_list.append(item)

print(unique_list)

Output:

[1, 2, 3, 4, 5]

This method preserves the original order of the list.

3. Using a Dictionary (or dict.fromkeys() method)

In Python 3.7+, dictionaries maintain insertion order, so using a dictionary is a good way to preserve the order of elements while removing duplicates.

You can use the fromkeys() method to create a dictionary from the list, where the list elements become dictionary keys. Since dictionary keys are unique, this removes duplicates.

Example:

# Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates using dict.fromkeys()
unique_list = list(dict.fromkeys(my_list))

print(unique_list)

Output:

[1, 2, 3, 4, 5]

4. Using List Comprehension and a Set for Tracking

You can use list comprehension in combination with a set to efficiently track already seen elements and ensure only unique elements are added to the result list.

Example:

# Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates using list comprehension
seen = set()
unique_list = [item for item in my_list if not (item in seen or seen.add(item))]

print(unique_list)

Output:

[1, 2, 3, 4, 5]

This method preserves the order of elements and efficiently removes duplicates.

5. Using itertools.groupby()

If your list is sorted, you can use itertools.groupby() to group consecutive duplicate elements. This method only works if the list is already sorted or you sort it beforehand.

Example:

from itertools import groupby

# Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Sort the list and use groupby to remove duplicates
sorted_list = sorted(my_list)
unique_list = [key for key, _ in groupby(sorted_list)]

print(unique_list)

Output:

[1, 2, 3, 4, 5]

6. Using collections.OrderedDict()

In Python 3.6+, dictionaries preserve insertion order, but you can also use OrderedDict from the collections module for versions prior to 3.6. This method preserves the order of the list.

Example:

from collections import OrderedDict

# Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates using OrderedDict
unique_list = list(OrderedDict.fromkeys(my_list))

print(unique_list)

Output:

[1, 2, 3, 4, 5]

Summary of Methods

Conclusion

If maintaining the order is important, using dict.fromkeys(), OrderedDict, or list comprehension with a set are good options. If order doesn’t matter, converting the list to a set is the simplest and most efficient method.

Let me know if you need more help! 😊